Augmented visualization for a surgical robot using a captured visible image combined with a fluorescence image  and a captured visible image

ABSTRACT

An endoscope with an optical channel is held and positioned by a robotic surgical system. A capture unit captures (1) a visible first image at a first time and (2) a visible second image combined with a fluorescence image at a second time. An image processing system receives (1) the visible first image and (2) the visible second image combined with the fluorescence image and generates at least one fluorescence image. A display system outputs an output image including an artificial fluorescence image.

RELATED APPLICATION

This application is a continuation of application Ser. No. 14/332,684(filed Jul. 16, 2014, disclosing “AUGMENTED VISUALIZATION FOR A SURGICALROBOT USING A CAPTURED VISIBLE IMAGE COMBINED WITH A FLUORESCENCE IMAGEAND A CAPTURED VISIBLE IMAGE”), which is a continuation of U.S. patentapplication Ser. No. 12/165,194 (filed Jun. 30, 2008, disclosing“AUGMENTED STEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT USING ACAPTURED VISIBLE IMAGE COMBINED WITH A FLUORESCENCE IMAGE AND A CAPTUREDVISIBLE IMAGE”) and which claims the benefit of U.S. Provisional PatentApplication No. 61/048,179 (filed Apr. 26, 2008, disclosing “AUGMENTEDSTEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT”), each of which isincorporated herein by reference in its entirety.

This application may be related to the following commonly assigned andcommonly filed U.S. patent applications, each of which is incorporatedherein by reference in its entirety:

-   -   1. U.S. patent application Ser. No. 12/164,363 entitled        “AUGMENTED STEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT,”        naming as inventors, David D. Scott et al., filed on Jun. 30,        2008);    -   2. U.S. patent application Ser. No. 12/164,976 entitled        “AUGMENTED STEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT USING        A CAPTURED FLUORESCENCE IMAGE AND CAPTURED STEREOSCOPIC VISIBLE        IMAGES,” naming as inventors, David D. Scott et al., filed on        Jun. 30, 2008;    -   3. U.S. patent application Ser. No. 12/165,189 entitled        “AUGMENTED STEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT USING        A CAMERA UNIT WITH A MODIFIED PRISM,” naming as inventors,        David D. Scott et al., filed on Jun. 30, 2008; and    -   4. U.S. patent application Ser. No. 12/265,189 entitled        “AUGMENTED STEREOSCOPIC VISUALIZATION FOR A SURGICAL ROBOT USING        TIME DUPLEXING,” naming as inventors, David D. Scott et al.,        filed on Jun. 30, 2008).

BACKGROUND 1. Field of Invention

Aspects of this invention are related to endoscopic imaging, and aremore particularly related to blending visible and alternate images so asto provide an enhanced real-time video display for a surgeon.

2. Art

The da Vinci® Surgical System, manufactured by Intuitive Surgical, Inc.,Sunnyvale, Calif., is a minimally invasive, teleoperated robotic systemthat offers patients many benefits, such as reduced trauma to the body,faster recovery and shorter hospital stay. One key component of the daVinci® Surgical System is a capability to provide two-channel (i.e.,left and right) video capture and display of three-dimensional (3D)visible images that provides stereoscopic viewing for the surgeon.

Such electronic stereoscopic imaging systems may output high definitionvideo images to the surgeon, and may allow features such as zoom toprovide a “magnified” view that allows the surgeon to identify specifictissue types and characteristics, as well as to work with increasedprecision. In a typical surgical field, however, certain tissue typesare difficult to identify, or tissue of interest may be at leastpartially obscured by other tissue.

SUMMARY OF THE INVENTION

In one aspect, a robotic surgical system positions and holds anendoscope. A visible imaging system is coupled to the endoscope. Thevisible imaging system captures a visible image of tissue. An alternateimaging system is also coupled to the endoscope. The alternate imagingsystem captures a fluorescence image of at least a portion of thetissue. A stereoscopic video display system is coupled to the visibleimaging system and to the alternate imaging system. The stereoscopicvideo display system outputs a real-time stereoscopic image comprising athree-dimensional presentation of a blend of a fluorescence imageassociated with the captured fluorescence image, and the visible image.

Thus, the stereoscopic video capturing and viewing capability ofsurgical robots is augmented by incorporating both stereoscopic visibleimages and stereoscopic alternate imaging modality images to identify,in real-time during surgery, tissue of clinical interest.

Aspects of the invention simultaneously provide stereoscopic alternatemodality images that identify tissue of clinical interest in addition tostereoscopic visible images that a surgeon normally uses when performinga surgical operation using a teleoperated surgical system. Thiscombination of stereoscopic visible and alternate images providesbenefits including, but not limited to, allowing a surgeon in real-timeto identify positive tumor margins for diseased tissue excision and toidentify nerves so as to avoid cutting those nerves.

This imaging combination may be a continuous overlay of the stereoscopicvisible and alternate images, or the overlay of stereoscopic alternateimages may be toggled on and off. Also, the real-time three-dimensionalblend of the visible image and the another fluorescence image ispresented in only one eye of the stereoscopic image in one aspect.

In another aspect, the visible imaging system captures the visible imageat a first frame rate, while the alternate imaging system captures thefluorescence image at a second frame rate. The first frame rate isdifferent from the second frame rate. The alternate imaging systemprovides fluorescence images to the stereoscopic video display system atthe first frame rate by generating artificial fluorescence images tosynchronize the fluorescence images with the visible images.

Thus, in one aspect, a method includes capturing, from an endoscope heldby and positioned by a robotic manipulator arm of a robotic surgicalsystem, a visible image of tissue. This method also captures, from theendoscope, an alternate image of at least a portion of the tissue. Thealternate image comprises a fluorescence image. In this method, a blendof another fluorescence image associated with the captured fluorescenceimage and the visible image are output in a real-time stereoscopic videodisplay.

In another aspect, the method generates a second fluorescence imageusing information associated with the captured fluorescence image. Thesecond fluorescence image is the another fluorescence image.

In still another aspect, the method generates a second visible imageusing information associated with the visible image. The visible imageand the second visible image comprise a stereoscopic pair of visibleimages. In this aspect, the method also generates a second fluorescenceimage using information associated with the fluorescence image. Thesecond fluorescence image is the another fluorescence image.

In one aspect, an illumination channel is held and positioned by arobotic surgical system. Light from the illumination channel illuminatestissue. A stereoscopic optical channel is also, held and positioned bythe robotic surgical system. The stereoscopic optical channel transportsfirst light from the tissue. Another optical channel also is held andpositioned by the robotic surgical system. This optical channeltransports second light from the tissue. The stereoscopic opticalchannel is different from the another optical channel.

An image capture system includes a first capture unit coupled to thestereoscopic optical channel. The first capture unit captures astereoscopic visible image from the first light. The image capturesystem also includes a second capture unit coupled to the anotheroptical channel. The second capture unit captures a fluorescence imagefrom the second light.

An intelligent image processing system is coupled to the first captureunit and to the second capture unit. The intelligent image processingsystem receives the captured stereoscopic visible image and the capturedfluorescence image. The intelligent image processing system generates astereoscopic pair of fluorescence images.

An augmented stereoscopic display system is coupled to the intelligentimage processing system, and to the image capture system. The augmentedstereoscopic display system outputs a real-time stereoscopic imagecomprising a three-dimensional presentation of a blend of thestereoscopic visible image and the stereoscopic pair of fluorescenceimages.

In another aspect, a method includes capturing a stereoscopic visibleimage of tissue from a stereoscopic optical path held and positioned bya robotic surgical system. This method also captures a fluorescenceimage of the tissue from another optical channel held and positioned bythe robotic surgical system. The stereoscopic optical channel isdifferent from the another optical channel.

The method processes the captured fluorescence image using informationfrom the captured stereoscopic visible image to generate a stereoscopicpair of fluorescence images. A real-time augmented stereoscopic image ofthe tissue comprising a three-dimensional presentation of a blend of thestereoscopic visible image and the stereoscopic pair of fluorescenceimages is generated.

In one aspect, an endoscope is held and positioned by a robotic surgicalsystem. The endoscope includes a stereoscopic optical channel, which hasa first channel for transporting first light from tissue and a secondchannel for transporting second light from the tissue.

A first capture unit is coupled to the first channel. The first captureunit captures: a visible first color component of a visible left imagecombined with a fluorescence left image from the first light; a visiblesecond color component of the visible left image from the first light;and a visible third color component of the visible left image from thefirst light.

A second capture unit is coupled to the second channel. The secondcapture unit captures: a visible first color component of a visibleright image combined with a fluorescence right image from the secondlight; a visible second color component of the visible right image fromthe second light; and a visible third color component of the visibleright image from the second light. The two capture units are included inan image capture system.

An augmented stereoscopic display system is coupled to the image capturesystem. The augmented stereoscopic display system outputs a real-timestereoscopic image of at least a portion of the tissue. The real-timestereoscopic image includes a three-dimensional presentation includingthe visible left and right images and the fluorescence left and rightimages.

The first capture unit includes a prism. The prism separates the firstlight into (1) the visible first color component of the visible leftimage, (2) the visible second color component of the visible left image,(3) the third color component visible left image, and (4) a fourthcomponent separated and removed from the first, second, and third colorcomponents and having a color of the first color component wherein thefourth component is the fluorescence left image. The second capture unitincludes a similar prism in one aspect.

In still yet another aspect, a method captures a visible first colorcomponent of a visible left image of tissue combined with a fluorescenceleft image of at least a portion of the tissue from a stereoscopicoptical path in an endoscope held and positioned by a robotic surgicalsystem. The method also captures a visible second color component of thevisible left image from the stereoscopic optical path; a visible thirdcolor component of the visible left image from the stereoscopic opticalpath; a visible first color component of a visible right image of thetissue combined with a fluorescence right image of at least a portion ofthe tissue from the stereoscopic optical path in an endoscope held andpositioned by a robotic surgical system; a visible second colorcomponent of the visible right image from the stereoscopic optical path;and a visible third color component of the visible right image from thestereoscopic optical path.

The method generates a real-time augmented stereoscopic image of thetissue. The real-time augmented stereoscopic image includes athree-dimensional presentation including the visible left and rightimages and the fluorescence left and right images.

This method uses a prism to separate light from the stereoscopic opticalpath into (1) the visible first color component, (2) the visible secondcolor component, (3) the visible third color component, and (4) a fourthcomponent separated and removed from the first, second, and third colorcomponents and having a color of the first color component. The fourthcomponent is the fluorescence image.

In one aspect, an endoscope also is held and positioned by a roboticsurgical system. The endoscope includes a stereoscopic optical channelfor transporting light from tissue. A capture unit is coupled to thestereoscopic optical channel. The capture unit captures (1) a visiblefirst image and (2) a visible second image combined with a fluorescencesecond image from the light. The first image is one of a left image anda right image. The second image is the other of the left image and theright image.

An intelligent image processing system is coupled to the capture unit toreceive (1) the visible first image and (2) the visible second imagecombined with the fluorescence second image. The intelligent imageprocessing system generates at least one fluorescence image of astereoscopic pair of fluorescence images and a visible second image.

An augmented stereoscopic display system is coupled to the intelligentimage processing system, and to the image capture system. The augmentedstereoscopic display system outputs a real-time stereoscopic imageincluding a three-dimensional presentation. The three-dimensionalpresentation includes in one eye, a blend of the at least onefluorescence image of a stereoscopic pair of fluorescence images and oneof the visible first and second images; and in the other eye, the otherof the visible first and second images.

In yet a further aspect, a method captures a visible first image oftissue from a stereoscopic optical path in an endoscope held andpositioned by a robotic surgical system. The method also captures avisible second image combined with a fluorescence second image of thetissue from the stereoscopic optical path in the endoscope held andpositioned by the robotic surgical system. The first image is one of aleft image and a right image. The second image is the other of the leftimage and the right image.

The method processes the visible first image and the visible secondimage combined with the fluorescence second image to generate at leastone fluorescence image of a stereoscopic pair of fluorescence images anda visible second image. A real-time stereoscopic image comprising athree-dimensional presentation is generated. The three-dimensionalpresentation includes: in one eye, a blend of the at least onefluorescence image of a stereoscopic pair of fluorescence images and oneof the visible first and second images; and in the other eye, an otherof the visible first and second images.

In one aspect, an endoscope is again held and positioned by a roboticsurgical system. The endoscope includes a stereoscopic optical channelfor transporting light from tissue. A capture unit is coupled to thestereoscopic optical channel.

The capture unit captures (1) at a first time, a first image from thelight; and (2) at a second time different from the first time, a secondimage from the light. Only one of the first image and the second imageincludes a combination of a fluorescence image and a visible image. Theother of the first image and the second image is a visible image.

An intelligent image processing system is coupled to the capture unit.The intelligent image processing system generates an artificialfluorescence image using the captured fluorescence image. An augmentedstereoscopic display system is coupled to the intelligent imageprocessing system. The augmented stereoscopic display system outputs anaugmented stereoscopic image of at least a portion of the tissuecomprising the artificial fluorescence image.

In one aspect, the fluorescence image includes a fluorescence left imageand a fluorescence right image. The first image comprises a stereoscopicpair of images including: a visible left image combined with thefluorescence left image: and a visible right image combined with thefluorescence right image. The robotic surgical system generates anartificial stereoscopic pair of fluorescence images for the second timeusing the fluorescence left and right images so that the artificialstereoscopic pair of fluorescence images are the artificial fluorescenceimage. The intelligent image processing system also includes temporalimage registration for registering the first image and the second image.

In another aspect, the first image includes a visible image which inturn includes a visible first color component, a visible second colorcomponent, and a visible third color component. The second imageincludes a visible image combined with a fluorescence image including: avisible first color component combined with the fluorescence image, avisible second color component and a visible third color component. Theintelligent image processing system further comprises a fluorescenceimage and artifacts generator to generate (1) artifacts for the visiblesecond and third color components, and (2) the fluorescence image plusartifacts for the visible first color component.

In this aspect, the intelligent image processing system also includes afluorescence image extractor coupled to the fluorescence image andartifacts generator. The fluorescence image extractor generates a firstfluorescence image for the second time. A fluorescence image enhancementsystem is coupled to the fluorescence image generator. The fluorescenceimage enhancement system receives the first fluorescence image andgenerates the artificial fluorescence image.

In still yet a further aspect, a method includes capturing at a firsttime, a first image from light from a stereoscopic optical path in anendoscope held and positioned by a robotic surgical system at a firsttime wherein the light is from tissue. This method also includescapturing at a second time different from the first time, a second imagefrom the light wherein only one of the first image and the second imageincludes a combination of a fluorescence image and a visible image; andan other of the first image and the second image is a visible image. Anartificial fluorescence image is generated using the capturedfluorescence image. An augmented stereoscopic image of at least aportion of the tissue including the artificial fluorescence image isalso generated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view of an augmented stereoscopic visualizationsystem for a minimally invasive surgical robot.

FIG. 2 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 1.

FIG. 3A is a schematic view that illustrates hardware and software(image processing and user interface) aspects of the use of two separateoptical paths (but one camera unit for fluorescence imaging) forcapturing, processing, and outputting blended real-time stereoscopicvisible and fluorescence images in a minimally invasive surgical roboticsystem.

FIG. 3B is a more detailed view showing endoscopes with two separateoptical paths, and separate camera units coupled to each optical path.

FIG. 3C illustrates one aspect of a combination illumination sourceconnected to a fiber optic cable.

FIG. 3D illustrates one aspect of a spatial image registration system.

FIG. 3E is an alternate schematic view that illustrates hardware andsoftware (image processing and user interface) aspects of the use of twoseparate optical paths and stereo cameras for capturing, processing, andoutputting blended real-time stereoscopic visible and fluorescenceimages in a minimally invasive surgical robotic system.

FIG. 4 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 3A.

FIG. 5A is a schematic view that illustrates hardware and software(image processing and user interface) aspects of the use of a singlestereoscopic optical path with separate cameras for capturing,processing, and outputting blended stereoscopic real-time visible andfluorescence images in a minimally invasive surgical robotic system.

FIG. 5B is a more detailed view showing an endoscope two separate cameraunits coupled to the endoscope.

FIGS. 5C to 5E and 5G illustrate aspects of the combination light sourceand aspects of a fiber optic bundle or bundles used to transport lightfrom the combination light source.

FIG. 5F illustrates one aspect for separating visible and fluorescenceimages from tissue.

FIG. 6 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 5A.

FIG. 7A is a schematic view that illustrates hardware and software(image processing and user interface) aspects of the use of channeldivision with a single stereoscopic optical path for capturing,processing, and outputting blended real-time stereoscopic visible andfluorescence images in a minimally invasive surgical robotic system.

FIG. 7B is a more detailed view showing an endoscope with a singlecamera unit coupled to the endoscope.

FIG. 8 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 7A.

FIGS. 9A and 9B are a schematic view that illustrates hardware andsoftware (image processing and user interface) aspects of the use oftime division with a single stereoscopic optical path for capturing,processing, and outputting blended stereoscopic visible and fluorescenceimages in a minimally invasive surgical robotic system.

FIG. 9C illustrates one aspect of the timing, synchronization, andcapture of the system in FIGS. 9A and 9B.

FIG. 9D is a schematic view that illustrates hardware and software(image processing and user interface) aspects of the use of timedivision with a single stereoscopic optical path for capturing,alternative processing, and outputting blended stereoscopic visible andfluorescence images in a minimally invasive surgical robotic system.

FIG. 9E is a schematic view that illustrates an alternative aspect ofthe intelligent image processing system.

FIG. 10A is a process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 9A.

FIG. 10B illustrates aspects of the timing, synchronization, capture,and artificial fluorescence frames generated using the process of FIG.10A.

FIG. 11A is a schematic view that illustrates hardware and softwareaspects of the use of time division and capturing a fluorescence imagewith one of the visible color components using a single stereoscopicoptical path, processing, and outputting blended stereoscopic visibleand fluorescence images in a minimally invasive surgical robotic system.

FIG. 11B illustrates one aspect of the timing, synchronization, andcapture of the system in FIG. 11A.

FIG. 12 is a process flow diagram for one aspect of the intelligentimage processing system of FIG. 11A.

FIG. 13A is a schematic view that illustrates hardware and software(image processing and user interface) aspects of using a singlestereoscopic optical path, capturing with a camera unit having modifiedprisms, processing, and outputting real-time stereoscopic visible andfluorescence images in a minimally invasive surgical robotic system.

FIG. 13B illustrates a spectrum from a prism that separates visible andfluorescence light from tissue into a first color component of thevisible image, a second color component of the visible image, a thirdcolor component of the visible image, and a fourth component separatedand removed from the first, second, and third color components with thefourth component having a color of one the color components.

FIG. 14 is process flow diagram of a process performed using, forexample, the augmented stereoscopic visualization system for a minimallyinvasive surgical robot of FIG. 13A.

FIG. 15 illustrates one aspect of the timing, synchronization, andcapture for the time and channel division in an augmented stereoscopicvisualization system for a minimally invasive surgical robot utilizing astereoscopic endoscope with 1-chip CCD sensor.

In the drawings, the first digit of a figure number for single digitfigure numbers and the first two digits of a figure number for doubledigit figure numbers indicates the figure in which the element with thatfigure number first appeared.

As used herein, “robot” should be broadly construed, and includestelerobotic systems.

As used herein, electronic stereoscopic imaging includes the use of twoimaging channels (i.e., channels for left and right images).

As used herein, a stereoscopic optical path includes two channels in anendoscope for transporting light from tissue (e.g., channels for leftand right images). The light transported in each channel represents adifferent view of the tissue and so is sometimes referred to as firstlight and second light to distinguish the light in the two channels. Thelight can include one or more images.

As used herein, an illumination path includes a path in an endoscopeproviding illumination to tissue.

As used herein, images captured in the visible spectrum are referred toas visible images.

As used herein, images, not including visible images, captured in analternate imaging modality are referred to as alternate images. Anexample of an alternate imaging modality is an image that capturestissue fluorescence.

As used herein, images captured as the result of fluorescence arereferred to herein as fluorescence images. There are variousfluorescence imaging modalities. Fluorescence may result from the useof, e.g., injectable dyes, fluorescent proteins, or fluorescent taggedantibodies. Fluorescence may result from, e.g., excitation by laser orother energy source. Fluorescence images can provide vital in vivopatient information that is critical for surgery, such as pathologyinformation (e.g., fluorescing tumors) or anatomic information (e.g.,fluorescing tagged nerves).

As used herein, a long pass filter lets all the wavelengths longer thana wavelength number through. For instance, a 510 nm long pass filterlets all the wavelengths greater than 510 nm through the filter.Typically, long pass filters are used as barrier filters influorescence. A long pass filter is sometimes used to pass thefluorescence light through the filter and not the excitation light.

As used herein, a short pass filter lets light through the filter thatis lower in wavelength than a wavelength of the filter. For instance, a480 nm short pass filter lets light that is shorter in wavelength than480 nm (less than 480 nm) through the filter. Short pass filters aresometimes used as excitation filters for fluorescence.

As used herein, a band pass filter allows only a set of wavelengthsthrough. The wavelength number is referred to as the center wavelengthof a band pass filter. The center wavelength is the wavelength thatallows the most light through within the range of wavelengths that willbe passed through the filter. Frequently this is the center wavelengthof the filter. A band pass filter is rated by center wavelength and thepass band or width.

DETAILED DESCRIPTION

Aspects of this invention augment the stereoscopic video capturing andviewing capability of surgical robots, e.g., the da Vinci® SurgicalRobot System manufactured by Intuitive Surgical, Inc. of Sunnyvale,Calif. by incorporating both stereoscopic visible images andstereoscopic alternate imaging modality images to identify, in real-timeduring surgery, tissue of clinical interest. (da Vinci® is a registeredtrademark of Intuitive Surgical, Inc. of Sunnyvale, Calif.)

Aspects of the invention simultaneously provide stereoscopic alternatemodality images that identify tissue of clinical interest in addition tostereoscopic visible images that a surgeon normally uses when performinga surgical operation using a teleoperated surgical system. Thiscombination of stereoscopic visible and alternate images providesbenefits including, but not limited to, allowing a surgeon in real-timeto identify positive tumor margins for diseased tissue excision and toidentify nerves so as to avoid cutting those nerves.

This imaging combination may be a continuous overlay of the stereoscopicvisible and alternate images, or the overlay of stereoscopic alternateimages may be toggled on and off (e.g., by using a foot pedal or bydouble-clicking master finger grips on the da Vinci® Surgical Systemsurgeon's console).

FIG. 1 is a high level diagrammatic view of one robotic surgical system,for example, the da Vinci® Surgical System, including an augmentedstereoscopic visualization system 100. In this example, a surgeon, usinga surgeon's console 114, remotely manipulates an endoscope 112 using arobotic manipulator arm 113. There are other parts, cables etc.associated with the da Vinci® Surgical System, but these are notillustrated in FIG. 1 to avoid detracting from the disclosure.

As explained more completely below, an illumination system (not shown)is coupled to endoscope 112. Typically, the illumination system provideswhite light and at least one fluorescence excitation light. All or partof this light is coupled to at least one illumination path in endoscope112 by a fiber optic bundle. The light passes through at least oneillumination path in endoscope 112 and illuminates tissue 103 of apatient 111. Endoscope 112 also includes, in one aspect, two opticalchannels for passing light from the tissue, e.g., reflected white lightand fluorescence. The reflected white light is a visible image, whilethe fluorescence is a fluorescence image.

The white light reflected from tissue 103 is captured as visiblestereoscopic images 121 in image capture system 120. Similarly, afluorescence image or fluorescence images 122 are also captured in imagecapture hardware 120. As explained more completely below, there are avariety of ways that the various images needed for the stereoscopicdisplay can be captured. Typically, image capture hardware 120 includesat least one camera including a charge-coupled device (CCD) sensor. Thecapture of the images occurs simultaneously or nearly simultaneously inimage capture system 120.

In one aspect, intelligent image processing system 130 functions incooperation with image capture system 120 to extract the fluorescenceimage from the information provided from the optical channel. Forexample, filter processing is working with a spectrum balancer tocompensate for any degradation to the visible image in the process ofremoving the fluorescence image given the frequency of the laser lightused to excite the fluorescence.

Also, the captured images are processed for subsequent displaystereoscopically in intelligent imaging processing system 130. Forexample, when separate optical channels having a fixed relationship areused for transporting the fluorescence image and the reflected whitelight image to intelligent image processing system 130, a one stepcalibration is used based upon the fixed relative positions of theseparate optical channels.

Intelligent image processing system 130, also when appropriate, performsspatial image registration of the fluorescence image(s) and the visibleimages. The spatial image registration permits proper overlay of thefluorescence image in augmented stereoscopic display system 140.

In another aspect, intelligent image processing system 130 does stereomatching of left-channel and right-channel visible images. In stillother aspects, intelligent image processing generates artificial visibleand/or fluorescence images so that an augmented stereoscopic display canbe presented to the surgeon.

Thus, for a surgeon to perform minimally invasive surgery using surgicalrobotic system with augmented stereoscopic visualization 100, tissue 103is illuminated in an illuminate tissue process 201 (FIG. 2) to allowcapture of both visible images and alternate images, such asfluorescence images.

Knowledgeable individuals understand that fluorescence can occurnaturally when tissue itself is excited by a particular wavelengthlight, or alternatively, when tissue-specific fluorophores attached totissue 103 are excited by a particular wavelength of light. Thus, thefluorescence images described herein can be obtained by eithertechnique. Knowledgeable persons also know that some fluorophores emitenergy within the visible spectrum, and others emit energy outside thevisible spectrum (e.g., at approximately 830 nm).

Aspects of the invention include illumination of tissue using both abroad spectrum white light source for visible images and another lightsource for the alternate images in illuminate tissue process 201. Forexample, narrow band light to excite tissue-specific fluorophores may beused as the light source for the alternate images.

For fluorescence alternate images, if the excitation wavelength occursin the visible spectrum, the white light may function to excite thefluorophores. If the excitation wavelength occurs outside the visiblespectrum (e.g., in the near infrared (IR)) or if additional excitationenergy is required at a wavelength in the visible spectrum, a lasermodule (or other energy source, such as a light-emitting diode orfiltered white light) is used to simultaneously illuminate the tissue inilluminate tissue process 201. This simultaneous illumination can beaccomplished in various ways as discussed more completely below.

The light from tissue 103, reflected and emitted, is conditioned inpre-process light from tissue process 202. For example, the light isfiltered to enhance the contrast in the images in the stereoscopic videodisplay. If the reflected and emitted light are included in a singlelight channel, pre-process light from tissue process 202 separates thelight into reflected light and emitted light.

Pre-process light from tissue process 202 is optional and may not beused in some aspects. Thus, either the preprocessed light from tissue103 or the original light from tissue 103 is passed to capture imagesprocess 203.

The output from pre-process light from tissue process 202 is captured incapture images process 203 as visible images and alternate image(s). Seefor example, image capture system 120 described above.

Intelligent processing 204 performs the necessary processes on thecaptured image to provide a complete set of visible and fluorescentimages for stereoscopic display.

In generate stereoscopic video display of tissue process 205, the set ofvisible and fluorescent images are blended as needed to generate a threedimensional presentation of the tissue. The three-dimensionalpresentation eliminates problems in the prior art associated withvarying distances and geometries of the tissue with respect to theendoscope. In particular, the real-time stereoscopic display of thetissue with the alternate image and/or visible images provides anaccurate three-dimensional view of the tissue that the surgeon can usein determining the scope of the surgery, e.g., location of diseasedtissue, location of nerves or other organs etc.

Two Separate Optical Paths from Tissue

In the embodiment of FIG. 3A, a robotic surgical system (not shown)includes two separate and distinct optical paths for transporting lightfrom tissue 303 to augmented stereoscopic vision system 300. Light fromthe two optical paths is used to generate a real-time stereoscopic videodisplay of tissue 303 for the surgeon operating the robotic surgicalsystem.

In one aspect, the stereoscopic video display includes a normalthree-dimensional view of tissue 303 augmented with one or morealternate images to highlight regions of interest in the tissue such asdiseased portions of tissue 303 and/or a specific tissue, such as anerve or organ different from that being operated on. Typically, thealternate image is presented in a specific color, e.g., blue.

In this example, two separate endoscopes 301, 302 are shown as providinga stereoscopic optical path and at least one other optical path fromtissue 303 to hardware 320. Endoscope 302 includes two light channelsthat make up the stereoscopic optical path, while endoscope 301 includesat least one light channel. Alternatively, all of the light channels canbe in a single endoscope. Accordingly, the aspects of FIG. 3A areillustrative only and are not intended to limit this embodiment to thespecific aspects shown.

In this example, endoscopes 301 and 302 each include an illuminationpath for providing light from combination light source 310 to tissue303. Alternatively, a single illumination path could be used to providethe light to tissue 303. While it is not shown, in one aspect,endoscopes 301 and 302 are each held and moved by the robotic surgicalsystem in a fixed relationship. See FIG. 1 for example. Alternatively,different robotic arms could be used to hold and move the two endoscopesseparately. In such an aspect, the real time kinematic information fromthe robotic arms is used in aligning the two endoscopes.

In this example, augmented stereoscopic vision system 300 includes acombination light source 310, hardware 320, and a plurality ofcomputer-based methods 390. As shown in FIG. 3A, a portion of hardware320 makes up image capture system 120A. Another portion of hardware 320and a portion of plurality of computer-based methods 390 make upintelligent image processing system 130A. Yet another portion ofhardware 320 and another portion of plurality of computer-based methods390 make up augmented stereoscopic display system 140A. Within imagecapture system 120A and intelligent image processing system 130A, theportions that process visible images make up a visible imaging systemwhile the portions that process fluorescence images make up an alternateimaging system.

Also, method 400 of FIG. 4 is implemented, in one aspect, usingaugmented stereoscopic vision system 300. As shown in FIG. 4, method 400includes a plurality of separate processes. Method 400 is oneimplementation of method 200 (FIG. 2).

In one aspect, hardware 320 includes at least two camera units 331, 332(FIG. 3B). One camera unit 332 includes two 3-chip charge-coupled device(CCD) high definition cameras and another camera 331 unit includes aone-chip CCD camera.

In this aspect, camera unit 331 is coupled to endoscope 301 by a filterblock 333 that includes a filter, as described more completely below forpreprocessing the light from endoscope 301. Similarly, camera unit 332is coupled to endoscope 302 by a filter block 334 that includes afilter, as described more completely below for preprocessing the lightfrom endoscope 302. In another aspect, the filters can be incorporatedin the camera units or alternatively cannot be used. Hardware 320 alsoincludes hardware circuits for performing the functions described morecompletely below. Plurality of computer-based methods 390 are, forexample, software executing on a computer processor.

Those of knowledge appreciate that computer-based methods can also beimplemented using hardware only, implemented partially in hardware andpartially in executable computer code, or implemented entirely inexecutable computer code. Similarly, the hardware described herein couldalso be implemented as computer-based methods or a combination ofhardware and computer-based methods. Accordingly, the characterizationused herein with respect to hardware and computer-based methods areillustrative only and are not intended to be limiting to the specificaspect described.

Two Separate Optical Paths—Illumination

Combination light source 310, 310A (FIGS. 3A and 3C) includes a whitelight source 312A and another light source 311A. Combination lightsource 310 is used in conjunction with an illumination path in anendoscope to perform illuminate tissue process 201A (FIG. 4). Whitelight source 312A provides white light, e.g., a first light, whichilluminates tissue 303. Other light source 311 provides light, e.g., asecond light, for exciting alternate images of tissue 303. For example,narrow band light from light source 311A is used to excitetissue-specific fluorophores so that the alternate images arefluorescence images of specific tissue within tissue 303.

For alternate images that are fluorescence images, if the fluorescenceexcitation wavelength occurs in the visible spectrum, white light source312A (FIG. 3B) may be used as both the white light source and as asource to excite the fluorophores. If the fluorescence excitationwavelength occurs outside the visible spectrum (e.g., in the nearinfrared (IR)) or if additional excitation energy is required at awavelength in the visible spectrum, a laser module 317 (or other energysource, such as a light-emitting diode or filtered white light) is usedto simultaneously illuminate tissue 303.

Thus, in one aspect, fluorescence is triggered by light from lasermodule 317. As an example, antibody agents, which were obtained fromMedarex, Inc., were excited using a 525 nm laser.

The particular alternate light source selected for combination lightsource 310A depends on the fluorophore or fluorophores used. Excitationand emission maxima of various FDA approved fluorescent dyes used invivo are presented in Table 1.

TABLE 1 Excitation Emission Fluorescent Dye maxima (nm) maxima (nm)Fluorscein 494 521 Indocyanine Green 810 830 Indigo Carmine 436 inalkaline 528 in alkaline solution solution Methylene Blue 664 682

Table 2 presents examples of common protein fluorophores used inbiological systems.

TABLE 2 Fluorescent Excitation Emission proteins/Fluorophore maxima (nm)maxima (nm) GFP 489 508 YFP 514 527 DsRed (RFP) 558 583 FITC  494** 518** Texas red  595**  615** Cy5  650**  670** Alexa Fluor 568  578** 603** Alexa Fluor 647  650**  668** Hoechst 33258 346 460 TOPRO-3 642661 **Approximate excitation and fluorescence emission maxima forconjugates.

Those knowledgeable in the field understand that a fluorophore can bebound to an agent that in turn binds to a particular tissue of thepatient. Accordingly, when a particular fluorophore is selected,combination light source 310A includes a light source that provideslight with the excitation maxima wavelength for that fluorophore. Thus,given the fluorophore or fluorophores of interest, appropriate lightsources can be included in combination light source 310, 310A.

The above examples in Tables 1 and 2 are illustrative only and are notintended to limit this aspect to the particular examples presented. Inview of this disclosure, an alternate imaging characteristic of thetissue can be selected and then an appropriate light source can beselected based upon the fluorescence or other alternate imagingcharacteristics being utilized.

In one aspect, white light source 312A of combination light source 310A(FIG. 3C) uses a Xenon lamp 314 with (1) an elliptic back reflector 314Aand (2) a long pass ultraviolet (W) filter coating 314B is used tocreate broadband white illumination light for visible images. The use ofa Xenon lamp is illustrative only and is not intended to be limiting.For example, a high pressure mercury arc lamp, other arc lamps, or otherbroadband light sources may be used.

Band pass filter 318 removes expected fluorescence emission wavelengthsfrom the white illumination light. This increases the contrast of thefluorescence image. The fluorescence image capture chip(s) are preventedfrom being saturated with reflected light from the entire tissue at thefluorescence wavelength.

The filtered white light from filter 318 is directed into a fiber opticbundle 316. Similarly, the light from laser module 317 is directed intofiber optic bundle 315. As illustrated in FIG. 3A, fiber optic bundles315 and 316 are two separate and distinct fiber optic bundles. However,in another aspect, bundles 315 and 316 may be different groups of fibersin a common fiber optic bundle.

Two Separate Optical Paths—Image Capture System 120A

The visible images from tissue 303 (FIG. 3A) are captured from onestereoscopic optical path in endoscope 302, and the fluorescence imageis captured from a separate monoscopic or stereoscopic optical path inendoscope 301. As noted above, while the separate optical paths areillustrated in two separate endoscopes in FIG. 3A, the separate opticalpaths may be in a single endoscope. An advantage of using two separateoptical paths is optical efficiency because there is no loss due to,e.g., aspects that use beam splitting, as described below.

The light from the stereoscopic optical path of endoscope 302, e.g., afirst light, is passed through a fluorescence excitation andfluorescence filter 334A to remove the fluorescence excitationwavelengths and the fluorescence, which leaves the visible left andright images. This helps to enhance the contrast between the visiblestereoscopic image and the fluorescence image and to improve the qualityof the visible image.

The filtered visible images from fluorescence excitation andfluorescence filter 334A are captured as a visible left image 336 in aleft CCD 332A and a visible right image 338 in a right CCD 332B. LeftCCD 332A captures red, green, and blue images for visible left image336. Similarly, right CCD 332B captures red, green, and blue images forvisible right image 338. Left CCD 332A and right CCD 332B can bemultiple CCDs with each CCD capturing a different color component; asingle CCD with a different region of that CCD capturing a particularcolor component, etc.

Moreover, herein use of a monochrome charge-coupled device (CCD) isillustrative only. Instead of a monochrome CCD, an intensifiedcharge-coupled device (ICCD), a charge injection device (CID), a chargemodulation device (CMD), a complementary metal oxide semiconductor imagesensor (CMOS) or an electron beam charge-coupled device (EBCCD) type ofsensor may also be used. Similarly, herein, a 3-chip CCD sensor is alsoillustrative and a color CMOS image sensor, or a three-CMOS color imagesensor assembly may also be used. These comments apply to the various1-chip CCD sensors and 3-chip sensors described herein and so are notrepeated with respect to each aspect of such sensors described herein.

The light from the optical path of endoscope 301 is passed through afluorescence band-pass filter 333A to remove all visible wavelengths butthe fluorescence image. Fluorescence image 335 from fluorescenceband-pass filter 333A is captured in CCD 331A, which in one aspect is asingle CCD sensor.

In this aspect, filters 333A and 334A perform pre-process light fromtissue operation 202A (FIG. 4). Capture images process 203A is performedby capturing the various images in the CCDs as just described.

Two Separate Optical Paths—Intelligent Image Processing System 130A

Since captured visible and fluorescence images originate from opticalpaths at different locations, the captured images are aligned usingimage processing methods. In this example, typically prior to using thecamera units in a normal setting, captured images, typically visibleimages, are provided to single-step calibration 391. If the physicalrelationship between the two optical paths is constant, image alignmentis done once in single-step calibration 391 prior to normal use, and thealignment information is then applied to all captured images (a fixedrelationship or “single-step” calibration). Single-step calibration 391determines the displacement necessary to bring the images from the twooptical paths into proper alignment.

In one aspect of single-step calibration 391, at least two factors areconsidered:

-   -   1) the intrinsic calibration of each endoscope and camera that        is important for the geometric aspect of imaging, e.g., a) focal        length, b) optical center, and c) lens distortion parameters;        and    -   2) the relative position and orientation of the two optical        paths.        A computer-based method for such a calibration may involve using        the two endoscopes and associated camera units to capture images        of a calibration pattern, for example, a checker-board pattern.        So long as the relationship between the optical paths remains        fixed, this calibration is valid. In this aspect, single-step        calibration 391 (FIG. 3A) is calibration process 405 (FIG. 4)        within intelligent imaging process 204A.

The results from single-step calibration 391 are supplied to spatialimage registration 392 (FIG. 3A) of registration process 406 (FIG. 4).Spatial image registration 392 also receives as inputs, each of capturedimages 335, 336, and 338. Briefly, spatial image registration registersthe images taken from different viewing angles so that any twocorresponding pixels from both images, based on the registrationresults, refer to the same scene point in the world.

One aspect of spatial image registration 392 is presented in FIG. 3D.Table 3 presents image modalities that are sometimes used in spatialimage registration 392.

TABLE 3 Input for matching Raw Gradient Image Image Modalities ImageImages Features Visible against Yes Yes Yes visible Visible against YesYes fluorescence

In spatial image registration 392A (FIG. 3D), two of the capturedimages, e.g., the fluorescence image and one of the visible images, areinput as Image 1 to pre-processing 370 and Image 2 to pre-processing371. Depending on the feature being used in matching the visible againstthe fluorescence, pre-processing 370, 371 generates the appropriateinformation. For example, for gradient images, the gradient of the rawimages along the X and Y directions is generated. Similarly, imagefeatures are obtained by pre-processing raw images. Many image featuresare available, for example, a histogram of image intensities in a localregion.

The results from pre-processing 370, 371 are supplied to matchingprocess 372. There are many methods available for matching process 372.A first example of matching is a normalized cross-correlation ofgradient images computed using all pixels in a small region surroundingthe pixel at location (x, y). Another example is mutual informationbased matching with the inputs being intensity histograms.

The output of matching process 372 is a displacement (dx, dy) that givesthe best matching score after moving the pixel at (x, y) from one inputimage to the other input image. If the displacement (dx, dy) isgenerated for all the pixels in the inputs, the result is called adisparity map consisting of two images of dx(x, y) and dy(x, y).

In this aspect, fluorescence image 335, sometimes referred to ascaptured fluorescence image 335 or stored fluorescence image 335, isregistered to visible left image 336, sometimes referred to as capturedvisible left image 336 or stored visible left image 336, in spatialregistration 392. Alternatively, two visible images 336 and 338 can beregistered first and then registered against fluorescence image 335.Similarly, fluorescence image 335 is registered to visible right image338, sometimes referred to as captured visible right image 338 or storedvisible right image 338, in spatial registration 392. In general,herein, an image that is shown in a CCD is sometimes referred to as acaptured image or a stored image.

The results from spatial image registration 392 are available to imagewarper 340. Image warper 340 also receives as input capturedfluorescence image 335. Using the spatial image registrationinformation, image warper 340 converts captured fluorescence image 335into a stereoscopic fluorescence pair, e.g., an artificial fluorescenceleft image 341 and an artificial fluorescence right image 342 for use ingenerating the stereoscopic display of the fluorescence image. Herein,artificial is used to refer that an image that is generated by hardware,software, or a combination of the two, and is in contrast to a capturedimage.

Specifically, image warper 340 uses the registration of fluorescenceimage 335 to visible left image 336 to warp fluorescence image 335 intofluorescence left image 341. Similarly, image warper 340 uses theregistration of fluorescence image 335 to visible right image 338 towarp fluorescence image 335 into fluorescence right image 342. Thus,image warper 340 performs generate stereoscopic fluorescence imagesprocess 407 (FIG. 4). Note that while this description is necessarilylinear and describes a single pass through the processing, the processesare occurring in real-time and the various images are being continuouslyupdated to reflect the current state of tissue 303 as observed viaendoscopes 301, 302.

Two Separate Optical Paths—Augmented Stereoscopic Display System 140A

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. For example, in a first mode, onlystereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, the fluorescence images aresuperimposed on the visible images to create augmented images, and thestereoscopic augmented images are output to the surgeon.

The video output may be toggled between these two modes by using, e.g.,a foot switch, a double click of the master grips that control thesurgical instruments, voice control, and other like switching methods.The toggle for switching between the two modes is represented in FIG. 3Aas display mode select 360.

In response to a user input 420 (FIG. 4), the signal from display modeselect 360 (FIG. 3A) is provided to a display mode check operation 408in a user interface 393 that in turn provides a control signal to blendcircuit 351 and blend circuit 352. If the surgeon selects visible only,visible left image 336 and visible right image 338 are presented instereoscopic display 350 via stereoscopic display of visible imageprocess 411 (FIG. 4) in generate stereoscopic video display of tissueprocess 205A. In one aspect, intelligent image processing system 130,can include spectrum balancers, similar to those shown in FIG. 5A, tocolor balance the visible left and right images provided to blendcircuit 351 and blend circuit 352, respectively.

If the surgeon selects visible plus fluorescence, in blend process 409,blend circuit 351 blends fluorescence left image 341 and visible leftimage 336, while blend circuit 352 blends fluorescence right image 342and visible right image 338. Different image blending options, such asalpha blending, can be implemented in blend circuits 351, 352. Theoutputs of blend circuits 351, 352 are presented in stereoscopic display350 via stereoscopic display of visible and fluorescence images process410 (FIG. 4).

Since fluorescence images 341, 342 show tissue of medical interest,fluorescence images 341, 342 can be processed to enhance the surgeon'svideo display presentation. This processing produces an artificialfluorescence image. For example, the fluorescing regions in thefluorescence image may be artificially colored (pseudo-colored) usingknown methods. When the artificial fluorescence image is blended withthe visible video image, the surgeon then sees the fluorescing tissue(e.g., artificially made bright green) in a high contrast to thesurrounding tissue in the visible image. Again, different image blendingoptions, such as alpha blending, of the pseudo color fluorescence imagesand visible images are made available.

In another aspect of enhancing the fluorescence image, a highly visibleborder is placed around the fluorescence area using known methods.Frequently, the fluorescing tissue is associated with a go or no godecision, e.g., remove or do not remove, by the surgeon and so thehighly visible border is of assistance.

In yet another aspect of enhancing the fluorescence images, a localhistogram equalization is performed on raw fluorescence image data 335.Instead of performing a histogram equalization for the entirefluorescence image frame, one or more local areas are identified aroundthe portion of the fluorescence image that shows the fluorescing tissue.The histogram equalization is performed on the one or more local areasto balance the light dark fluorescence appearance in the enhancedfluorescence image. Such image enhancement also helps spatial imageregistration.

Further, the fluorescence image may be artificially sustained in thevideo output to the surgeon. As an example, the fluorescence image maybe sustained after an injected agent no longer fluoresces so that thefluorescing region is still visible to the surgeon.

The stable platform provided by robotic surgical system, which holds theendoscope or endoscopes, facilitates the processing of the capturedfluorescence image in real-time because, unlike hand-held endoscopes, itis unnecessary to compensate for instability of the endoscope orendoscopes which typically results in blurred fluorescence images forhand guided endoscopes. In addition, the sharper fluorescence imagerelative to hand held endoscopes facilitates the enhanced processing ofthe captured fluorescence image.

Two Separate Optical Paths—Multiple Fluorescence Images

In another aspect of using two separate optical paths, multiplefluorescence images can be captured using augmented stereoscopic visionsystem 300A (FIG. 3E). System 300A is similar to system 300 and so onlythe differences are described. Elements with the same reference numeralare the same or equivalent elements.

In the embodiment of FIG. 3E, a robotic surgical system (not shown)includes two separate and distinct stereoscopic optical paths fortransporting light from tissue 303A to augmented stereoscopic visionsystem 300A. Light from the two light paths is used to generate areal-time stereoscopic video display of tissue 303 for the surgeonoperating the robotic surgical system. The stereoscopic video displayincludes a normal three-dimensional view of tissue 303 augmented withtwo alternate images to highlight regions of interest in tissue 303Asuch as diseased portions of tissue 303A and other tissue in tissue303A, such as a nerve or organ. Typically, the alternate images are eachpresented in a different specific color that typically contrasts withthe colors normally seen in the stereoscopic video display.

Again in this example, two separate endoscopes 301A, 302 are shown asproviding the two distinct stereoscopic optical paths from tissues 303Ato hardware 320A. Endoscope 302 has two light channels, and endoscope301A has two light channels. The two light channels in endoscope 301Aare used for capturing two different fluorescence images, fluorescenceimage 1 and fluorescence image 2. For convenience, fluorescence image 1is taken to be the same as the fluorescence image in FIG. 3A, and thevisible images are taken as the same visible images as in FIG. 3A. Inthis aspect, camera unit 331 includes at least a two-chip CCD camera.

Thus, the description of elements 333A, 331A, 335, 340, 341 and 342above are incorporated herein by reference. Similarly, the descriptionof elements 334A, 332A, 336, 332B, and 338 above are incorporated hereinby reference.

In this example, augmented stereoscopic vision system 300A includes acombination source illumination system 310C, hardware 320A, and aplurality of computer-based methods 390A. As shown in FIG. 3E, a portionof hardware 320A makes up image capture system 120B. Another portion ofhardware 320A and a portion of plurality of computer-based methods 390Amake up intelligent image processing system 130B. Yet another portion ofhardware 320A and another portion of plurality of computer-based methods390A make up augmented stereoscopic display system 140B

Combination light source 310C includes a white light source 312 and twoother light sources 311A and 311B. White light source 312 is similar tosource 312A (FIG. 3C) except in addition to filter 318 another filterremoves expected fluorescence emission wavelengths for fluorescenceimage 2 from the white illumination light.

Second light source 311A provides light to excite fluorescence image 1,while third light source 311B provides light to excite fluorescenceimage 2. In view of this disclosure, an appropriate light source can beselected based upon the fluorescence characteristics being utilized forthe two different fluorescence images.

In the aspect illustrated in FIG. 3E, one fiber optic bundle 316 coupleslight from white light source 312 to the illumination path in endoscope302. Fiber optic bundle 315A couples light from second light source 311Aand light from third light source 311B to the illumination paths inendoscope 315A. Specifically, a first set of fibers within fiber opticbundle 315A couples light from second light source 311A to a firstillumination path in endoscope 301A and a second set of fibers withinfiber optic bundle 315A couples light from third light source 311B to asecond illumination path in endoscope 301A.

This aspect is illustrative only and is not intended to be limiting. Forexample, if two illumination paths were in a single endoscope, acombination light source such as combination light source 510D withfiber optic bundle 514A (FIG. 5G) could be used instead of combinationlight source 310C.

Since the captured visible images and the two fluorescence imagesoriginated from optical paths at different locations, the capturedimages are aligned using image processing methods. In this example, thecaptured images are provided to single-step calibration 391A. Again, ifthe physical relationship between the two stereoscopic optical paths isconstant, image alignment is done once in single-step calibration 391A,and the alignment information is then applied to all captured images (afixed relationship or “single-step” calibration). The process insingle-step calibration 391A is equivalent to that described above foreach of the fluorescence images.

The results from single-step calibration 391A are supplied to spatialimage registration 392A (FIG. 3A) of registration process 406 (FIG. 4).Spatial image registration 392A also receives as inputs, each ofcaptured images 335, 335A, 336, and 338. The preprocessing and matchingdescribed above with respect to FIG. 3D is done for each of thefluorescence images. The results from spatial image registration 392 areavailable to image warpers 340, 340A.

Again, in one aspect, the augmented stereoscopic video output displaymay be operated in various modes. For example, in a first mode, onlystereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, the fluorescence images forfluorescence image 1 are superimposed on the visible images to createaugmented images, and the stereoscopic augmented images are output tothe surgeon. In a third mode, the fluorescence images for fluorescenceimage 2 are superimposed on the visible images to create augmentedimages, and the stereoscopic augmented images are output to the surgeon.In a fourth mode, the fluorescence images for fluorescence image 1 andthe fluorescence images for fluorescence image 2 are both superimposedon the visible images to create augmented images, and the stereoscopicaugmented images are output to the surgeon.

The video output may be toggled between these four modes by using, e.g.,a foot switch, clicks of the master grips that control the surgicalinstruments, voice control, and other like switching methods. The togglefor switching between the four modes is represented in FIG. 3E asdisplay mode select 360A.

In response to a user input 420 (FIG. 4) the signal from display modeselect 360A (FIG. 3A) is provided to user interface 393A that in turnprovides a control signal to blend circuit 351A and blend circuit 352A.If the surgeon selects visible only, visible left image 336 and visibleright image 338 are presented in stereoscopic display 350.

If the surgeon selects visible plus fluorescence image 1, blend circuit351A blends fluorescence left image 341 and visible left image 336,while blend circuit 352A blends fluorescence right image 342 and visibleright image 352. If the surgeon selects visible plus fluorescence image2, blend circuit 351A blends fluorescence left image 341A and visibleleft image 336, while blend circuit 352A blends fluorescence right image342A and visible right image 338. If the surgeon selects visible plusfluorescence image 1 plus fluorescence image 2, blend circuit 351Ablends fluorescence left image 341, fluorescence left image 341A andvisible left image 336, while blend circuit 352A blends fluorescenceright image 342, fluorescence right image 342A and visible right image338.

Again, the fluorescing regions in the fluorescence images may beartificially colored (pseudo-colored) using known methods. When theartificial fluorescence image is blended with the visible video image,the surgeon then sees the fluorescing tissue (e.g., artificially madebright green) in a high contrast to the surrounding tissue in thevisible image. Again, different image blending options, such as alphablending, of the pseudo color fluorescence images and visible images aremade available.

Single Stereoscopic Optical Path with Plurality of Cameras

In the embodiment of FIG. 5A, a robotic surgical system (not shown)includes a single stereoscopic optical path for transporting light fromtissue 503 to augmented stereoscopic vision system 500. Light from thesingle stereoscopic optical path is used to generate a real-timestereoscopic video display of tissue 503 for the surgeon operating therobotic surgical system. The stereoscopic video display includes athree-dimensional view of tissue 503 augmented with an alternate imageto highlight regions of interest in tissue 503 such as diseased portionsof tissue 503 and/or other tissue of interest, such as a nerve or organ.In one aspect, the alternate image is presented in a specific color,e.g., blue.

In this example, a single endoscope 501 provides the stereoscopicoptical path from tissue 503 to hardware 520. Endoscope 501 includes twolight channels that make up the stereoscopic optical path.

Endoscope 501 also includes an illumination path for providing light totissue 503. While it is not shown, endoscope 501 is held and moved bythe robotic surgical system. See FIG. 1 for example.

In this example, augmented stereoscopic vision system 500 includes acombination light source 510, hardware 520, and a plurality ofcomputer-based methods 590. As shown in FIG. 5A, a portion of hardware520 makes up image capture system 120C. Another portion of hardware 520and a portion of plurality of computer-based methods 590 make upintelligent image processing system 130C. Yet another portion ofhardware 520 and another portion of plurality of computer-based methods590 make up augmented stereoscopic display system 140C. Within imagecapture system 120C and intelligent image processing system 130C, theportions that process visible images make up a visible imaging systemwhile the portions that process fluorescence images make up an alternateimaging system.

Also, method 600 of FIG. 6 is implemented using augmented stereoscopicvision system 500. As shown in FIG. 6, method 600 includes a pluralityof separate processes. Method 600 is one implementation of method 200(FIG. 2).

In one aspect, hardware 520 includes at least two camera units 531, 532(FIG. 5B). One camera unit 532 includes a 3-chip charge-coupled device(CCD) high definition camera and at least a 1-chip CCD camera. Anothercamera 531 unit also includes a 3-chip charge-coupled device (CCD) highdefinition camera and at least a 1-chip CCD camera.

In this aspect, camera unit 531 and camera unit 532 are coupled toendoscope 501 by a block 533 that includes a filter and beam splitters,as described more completely below, for preprocessing the light fromendoscope 501. In another aspect, the filter can be incorporated in thecamera units.

Hardware 520 also includes hardware circuits for performing thefunctions described more completely below. Plurality of computer-basedmethods 590 are, for example, software executing on a computerprocessor.

The visible and fluorescence images are simultaneously captured via thesame stereoscopic optical path. One camera unit 531 captures visible andfluorescence left images, and second camera unit 532 captures visibleand fluorescence right images. In one aspect, camera units 531, 532 arelocked together.

Single Stereoscopic Optical Path with Plurality of Cameras—Illumination

Combination light source 510, 510A, 510B, 510C (FIGS. 5A, 5C, 5D, 5E)includes a white light source 512A and another light source 511A.Combination light source 510 is used in conjunction with an illuminationpath in endoscope 501 to perform illuminate tissue process 201B (FIG.6). White light source 512A provides light that illuminates tissue 503.Other light source 511A provides light for the alternate image of tissue503. For example, narrow band light from light source 511A is used toexcite tissue-specific fluorophores so that the alternate image is afluorescence image of specific tissue within tissue 503.

For alternate images that are fluorescence images, if the fluorescenceexcitation wavelength occurs in the visible spectrum, white light source512A (FIG. 5B) may be used as both the white light source and as asource to excite the fluorophores. If the fluorescence excitationwavelength occurs outside the visible spectrum (e.g., in the nearinfrared (IR)) or if additional excitation energy is required at awavelength in the visible spectrum, a laser module 517 (or other energysource, such as a light-emitting diode or filtered white light) is usedto simultaneously illuminate tissue 503.

In one aspect, white light source 512A is the same as white light source312A and the description of white light source 312A is incorporatedherein by reference.

In combination light source 510A (FIG. 5C), a small injection mirror 513is placed immediately in front of white light lamp unit 512A to reflectexcitation light through the focal point of white light lamp unit 512A.A turning mirror 516 is placed between laser module 517 and injectionmirror 513 to allow the optical path for the excitation light to bealigned with the white light. This mirror placement results in very highefficiency coupling of the white illumination light along with nearly100-percent efficiency of laser light coupling into fiber optic bundle514.

It has been observed that for the various aspects of the combinationlight sources, when the laser light is injected in a fiber optic bundle,the laser light disperses and illuminates tissue 503 adequately withoutrequiring any other dispersion techniques.

In combination light source 510B (FIG. 5D), a beam splitter 515 (e.g.,50/50 dichroic mirror; various beam splitting technologies are known) isused to incorporate both the white illumination light and laserexcitation light from turning mirror 516 into fiber optic bundle 514.

In another aspect (FIG. 5E), the white illumination light from whitelight source 512A and the laser excitation light from laser module 517are coupled together using a fiber optic bundle 514A in which severalfibers from fiber optic bundle 514A are split off and are separatelyterminated in a connector to which the laser light can be coupled.

In the case of the da Vinci® Surgical System, the endoscope has twoillumination paths. Thus, the fiber optic bundle is split so that twogroups of fibers, one carrying white light and the other carryingexcitation light are each directed into a different one of theillumination paths. An advantage of this aspect is that for existing daVinci® Surgical Systems, no excitation light alignment is required andvarious excitation light sources, as described herein, with variousexcitation light wavelengths can be easily swapped. For example, ifdifferent fluorophores with different excitation wavelengths are to beviewed during the same procedure (e.g., fluorophores associated with atumor and fluorophores associated with nearby nerves, such as inprostate surgery), the excitation lasers for the different fluorophorescan be easily exchanged in the combination light source. In one aspect,the fiber optic bundle or bundles remain connected to the combinationlight source while a light source is exchanged. Alternatively, two ormore excitation light sources can be coupled into the one or moreendoscope illumination channels in a similar manner.

In each of combination light sources 510, 510A, 510B, 510C, a band passfilter 518 removes expected fluorescence emission wavelengths from thewhite illumination light. This increases the contrast of thefluorescence image. The fluorescence image capture chip(s) are preventedfrom being saturated with reflected light from the tissue at thefluorescence wavelength.

Also, in one aspect, since charge-coupled devices (CCDs) are typicallysensitive at wavelengths outside the visible spectrum, a short passfilter 519 removes the unused IR wavelengths beyond the desired emissionand visible wavelengths. Removing the unused IR wavelengths increasesthe contrast for both the visible light image and the fluorescenceimage. In one embodiment the IR filter from the CCD cameras is removedto increase in sensitivity to the red and near IR wavelengths. Thefiltered white light is then directed into a fiber optic bundle, asdescribed above, and is coupled into the stereoscopic endoscope for usein illuminating tissue 503 for visible imaging.

Single Stereoscopic Optical Path with Plurality of Cameras—Image CaptureSystem 120C

A fluorescence right image λR and a visible image from tissue 503 (FIGS.5A, 5F) are transported in one path of the stereoscopic optical path inendoscope 501. Similarly, a fluorescence left image λL and a visibleleft image from tissue 503 are transported in the other path of thestereoscopic optical path in endoscope 501.

The images from the stereoscopic optical path of endoscope 501 arepassed through a fluorescence excitation filter 534 (FIG. 5F) to removethe fluorescence excitation wavelengths from the images. This helps toenhance the contrast between the visible stereoscopic image and thefluorescence image and to improve the quality of the visible image.

The filtered visible left image and the fluorescence left image interactwith a beam splitter 533A that splits the filtered images into a visibleleft image 536 that is captured in CCD 531A and a fluorescence leftimage 535 that is captured in CCD 531B. In one aspect, CCD 531A is a3-CCD sensor that captures the left RGB image and CCD 531B is a 1-CCDmonochromatic sensor that captures fluorescence left image 535.

Similarly, the filtered visible right image and the fluorescence rightimage interact with a beam splitter 533B that splits the filtered imagesinto a visible right image 538 that is captured in CCD 532A and afluorescence right image 537 that is captured in CCD 532B. In oneaspect, CCD 532A also is a 3-CCD sensor that captures the right RGBimage and CCD 532B is a 1-CCD monochromatic sensor that capturesfluorescence right image 537.

Thus, a total of four images—left and right visible and fluorescenceimages—are captured. An advantage of this aspect is that the alignmentbetween visible and fluorescence images is done in hardware as the chipsare physically positioned during manufacturing. In addition, the singleCCD can be selected for optimum sensing of the fluorescence image (e.g.,in near IR).

In this aspect, block 533 (FIG. 5F) is used to perform pre-process lightfrom tissue operation 202B (FIG. 6). Capture stereoscopic visible andstereoscopic fluorescence images process 203B is performed by capturingthe various images in the CCDs as just described.

Single Stereoscopic Optical Path with Plurality of Cameras—IntelligentImage Processing System 130C

Since the filtering described above creates a notch in each of visibleleft image 536 and visible right image 538, spectrum balancer 541, 542corrects the color balance for the notch. Color balancing is commonlyperformed in cameras and similar techniques are used herein. Forexample, the cameras can include a plurality of built-in color balances.Filtering processing 594 selects the correct built-in color balancebased upon the fluorescence filter characteristics for use in spectrumbalancer 541, 542. Alternatively, filter processing 594 in combinationwith spectrum balancer 541, 542 could implement the color balancingbased upon the fluorescence filter characteristics.

In this aspect, the combination of filter processing 594 and spectrumbalancer 541, 542 perform balance spectrum process 605 (FIG. 6) inintelligent image processing 204B.

Single Stereoscopic Optical Path with Plurality of Cameras—AugmentedStereoscopic Display System 140C

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. The operation of display mode select 560,user interface 593 and the interaction with blend circuit 551, 552 isthe same as the above description for display mode select 360, userinterface 393 and the interaction with blend circuit 351, 352 and thatdescription is incorporated herein by reference.

Thus, in response to a user input 620 (FIG. 6) the signal from displaymode select 560(FIG. 5A) is provided to a display mode check operation608 in a user interface 593 that in turn provides a control signal toblend circuits 551 and 552. If the surgeon selects visible only,spectrum balanced visible left image 536 and spectrum balanced visibleright image 538 are presented in stereoscopic display 550 viastereoscopic display of visible image process 611 (FIG. 6) in generatestereoscopic video display of tissue process 205B.

If the surgeon selects visible plus fluorescence, in blend process 609,blend circuit 551 blends fluorescence left image 535 and spectrumbalanced visible left image 536, while blend circuit 552 blendsfluorescence right image 537 and spectrum balanced visible right image538. Different image blending options, such as alpha blending, can beimplemented in blend circuits 551, 552. The outputs of blend circuits551, 552 are presented in stereoscopic display 550 via stereoscopicdisplay of visible and fluorescence images process 610 (FIG. 6). Thedisplayed fluorescence images can be processed in ways equivalent tothose described above with respect to FIGS. 3A to 4 and so are notrepeated here.

The above description of a camera unit with two cameras is illustrativeonly and is not intended to be limiting. For example, each camera unitcould be a single camera with optics that split the incoming beam. Inthis aspect, two chips of a 3-CCD image sensor are used to capture thevisible image, and the third chip is used to capture the fluorescenceimage. In this aspect, a prism (e.g., a trichroic beam splitter prismassembly) that directs light to the three CCD chips is designed suchthat the fluorescence wavelength light is reflected toward one CCD chipand the visible light is separated onto the other two CCD chips. Fullcolor for the visible images can be reconstructed from the two CCDchannels, as is commonly done. This aspect has the hardware alignmentadvantages described above.

In another aspect, features of intelligent image processing from FIG. 3Acan be combined with aspects of FIG. 5A. For example, combination lightsource 510D (FIG. 5G) includes white light source 512A and laser module517 configured, as described above with respect to FIG. 5D, to projectthe laser and white illumination light into one set of fibers in fiberoptic cable 514A. A second laser module 509 provides a beam that isinjected on a second set of fibers within fiber optic cable 514A. Thelight from two lasers 517, 509 excite different fluorescence emissionsand so each optical path in endoscope 501 includes a visible image andtwo fluorescence images.

In this aspect, beam splitter 533A is configured to separate the visibleleft image and the fluorescence left image for the first fluorescenceimage. Beam splitter 533B is configured to separate the visible rightimage and the fluorescence right image for the second fluorescenceimage.

Thus, in this aspect, the fluorescence left image of the firstfluorescence image is captured in CCD 531B and the fluorescence rightimage for the second fluorescence image is captured in CCD 532B. In eachcase, it is necessary to generate the other fluorescence image for thestereoscopic display.

The fluorescence left image of the first fluorescence image is spatiallyregistered with the visible right image and then an image warper isused, based on the registration, to generate the fluorescence rightimage for the first fluorescence image.

Similarly, the fluorescence right image of the second fluorescence imageis spatially registered with the visible left image and then an imagewarper is used, based on the registration, to generate the fluorescenceleft image for the second fluorescence image. Thus, the visible leftimage, visible right image, fluorescence left first image, fluorescenceright first image, fluorescence left second image, and fluorescenceright second image are available. Augmented stereoscopic display system140B of FIG. 3E is used to display the various images.

Thus, in general, for aspects in which the fluorescence imageinformation is captured in only one stereoscopic channel, a fluorescenceimage for the other channel must be generated to produce a stereoscopicfluorescence image display for the surgeon that is correlated to thestereoscopic visible image video. If the visible and fluorescence imagesshare the same optical path, stereo matching of visible images is usedto generate the fluorescence image for the second stereoscopic channel.If the visible and fluorescence images use different optical paths, thevisible and fluorescence images are registered to each other in thechannel that includes the captured fluorescence images, and then stereomatching of visible images is applied to generate the fluorescenceimages for the second channel.

Single Stereoscopic Optical Path with a Camera Unit

In one exemplary process, both visible and fluorescence images arecaptured in the right stereoscopic channel, and only a visible image iscaptured in the left channel. For example, in the embodiment of FIG. 7A,a robotic surgical system (not shown) includes a single stereoscopicoptical path for transporting light from tissue 703 to augmentedstereoscopic vision system 700. Light from the single stereoscopicoptical path is used to generate a real-time stereoscopic video displayof tissue 703 for the surgeon operating the robotic surgical system.

The stereoscopic video display includes a three-dimensional view oftissue 703 augmented with an alternate image to highlight regions ofinterest in tissue 503 such as diseased portions of tissue 503 and/orother tissue of interest, such as a nerve or organ. In one aspect, thealternate image is provided to only one eye, e.g., the right eye, in thestereoscopic view so that the surgeon can compare the left eye and righteye images without having to toggle between the augmented andnon-augmented stereoscopic view. In addition, this aspect also providesa stereoscopic view with a stereoscopic alternate view. This isaccomplished without the beam splitters used in FIG. 5a for example. Inone aspect, the alternate view is presented in a specific color, e.g.,blue.

In this example, a single endoscope 701 provides the stereoscopicoptical path from tissue 703 to hardware 720. Endoscope 701 has twolight channels making up the stereoscopic optical path and at least oneillumination channel for providing light to tissue 701. While it is notshown, endoscope 701 is held and moved by the robotic surgical system.See FIG. 1 for example.

In this example, augmented stereoscopic vision system 700 includes acombination light source 710, hardware 720, and a plurality ofcomputer-based methods 790. As shown in FIG. 7A, a portion of hardware720 makes up image capture system 120D. Another portion of hardware 720and a portion of plurality of computer-based methods 790 make upintelligent image processing system 130D. Yet another portion ofhardware 720 and another portion of plurality of computer-based methods790 make up augmented stereoscopic display system 140D. Within imagecapture system 120D and intelligent image processing system 130D, theportions that process visible images make up a visible imaging systemwhile the portions that process fluorescence images make up an alternateimaging system.

Also, method 800 of FIG. 8 is implemented using augmented stereoscopicvision system 700. As shown in FIG. 8, method 800 includes a pluralityof separate processes. Method 800 is one implementation of method 200(FIG. 2).

In one aspect, hardware 720 includes a single camera unit 731 (FIG. 7B).Camera unit 731 includes a 3-chip charge-coupled device (CCD) sensor foreach optical path of endoscope 701.

In this aspect, camera unit 731 (FIG. 7B) is coupled to endoscope 701 bya block 733 that includes a filter 733A (FIG. 7A) for preprocessing thelight from the left optical path of the stereoscopic optical path ofendoscope 701. In another aspect, the filter can be incorporated in thecamera unit. The visible right image with the fluorescence image and thevisible left image are simultaneously captured via the same stereoscopicoptical path.

Hardware 720 also includes hardware circuits for performing thefunctions described more completely below. Plurality of computer-basedmethods 790 are, for example, software executing on a computerprocessor.

Single Stereoscopic Optical Path with a Camera Unit—Illumination

Combination light source 710 with fiber optic bundle 714 is equivalentto any one of combination light sources 510A (FIG. 5C), 510B (FIG. 5D)and 510C (FIG. 5E) and the associated fiber optic bundles, as well asthe various aspects described above with respect to the implementationof combination light source 310A (FIG. 3C). Rather than repeat thedescription of those combination light sources that description isincorporated herein by reference. Combination light source 710 is usedin conjunction with an illumination path in endoscope 701 to performilluminate tissue process 201C (FIG. 8)

Single Stereoscopic Optical Path with a Camera—Image Capture System 120D

The visible left image from tissue 703 (FIG. 7A) is captured from a leftoptical channel of the stereoscopic optical path in endoscope 701, andthe visible right image combined with the fluorescence image from tissue703 is captured from a right optical channel of the stereoscopic opticalpath in endoscope 701.

To capture only the visible left image, the light from the left opticalchannel is filtered by fluorescence filter 733A to remove thefluorescence wavelength(s) from visible left image 736 that is capturedin a left CCD 731A. A visible and fluorescence right image 738 iscaptured in a right CCD 731B. Left CCD 731A captures red, green, andblue images for visible left image 731. Similarly, right CCD 731Bcaptures red, green, and blue images for visible and fluorescence rightimage 738.

In this aspect, filter 733A performs pre-process light from tissueoperation 202C (FIG. 8). Capture visible left image and visible andfluorescence right images process 203C is performed by capturing thevarious images in the CCDs as just described.

Single Stereoscopic Optical Path with a Camera Unit—Intelligent ImageProcessing System 130D

Spatial image registration 792 receives as inputs, each of capturedimages 736 and 738. Again, spatial image registration registers theimages taken from different viewing angles so that any two correspondingpixels from both images, based on the registration results, refer to thesame scene point in the world. Spatial image registration 792 isperformed in registration process 805 (FIG. 8) in intelligent processing204C.

In one aspect, spatial image registration 792, for the image modalitiesin Table 4, is the same as that presented in FIG. 3D.

TABLE 4 Input for matching Raw Gradient Image Image Modalities ImageImages Features Visible against Yes Yes Yes visible + fluorescence

In spatial image registration 392A (FIG. 3D), two of the capturedimages, e.g., a visible left image and a visible and fluorescence rightimage, are input as Image 1 to pre-processing 370 and Image 2 topre-processing 371. Depending on the feature being used in matching thevisible against the visible and fluorescence, pre-processing 370, 371generates the appropriate information. For example, for gradient images,the gradient of the raw images along the X and Y directions isgenerated. Similarly, image features are obtained by pre-processing rawimages. Many image features are available, for example, a histogram ofimage intensities in a local region.

The results from pre-processing 370, 371 are supplied to matchingprocess 372. There are many methods available for matching process 372.A first example is a normalized cross-correlation of either raw imagesor gradient images computed using all pixels in a small regionsurrounding pixel at location (x, y). Another example is mutualinformation based matching with the inputs being intensity histograms.

The output of matching process 372 is a displacement (dx, dy) that givesthe best matching score after moving the pixel at (x, y) from one inputimage to the other input image. If the displacement (dx, dy) isgenerated for all the pixels in the inputs, the result is called adisparity map consisting of two images of dx(x, y) and dy(x, y).

The pixel by pixel registration for the left and right images in spatialimage registration 792 is available to image warper 740 and image warper741. Image warper 740 also receives as input captured visible left image736.

Using the spatial image registration information and visible left image736, image warper 340 generates a visible right image and in turn, thevisible right image is supplied to image subtractor 743. Imagesubtractor 743 subtracts the visible right image from captured visibleand fluorescence image 738 to generate fluorescence right image 744.Using the spatial image registration information and fluorescence rightimage 744, image warper 741 generates a fluorescence left image 742.Image subtractor 745 subtracts fluorescence right image 744 fromcaptured visible and fluorescence image 738 to generate visible rightimage 746.

Thus, in this aspect, the combination of elements 792, 740, 743, 744,736, 738 and 745 are used in generate visible right image process 806(FIG. 8). The combination of elements 792, 740, 743, 736 and 738 areused to generate the fluorescence right image in generate fluorescenceleft and right images process 807, while the combination of elements792, 740, 743, 744, 741, 736 and 738 are used to generate thefluorescence left image in generate fluorescence left and right imagesprocess 807.

The processes described above are illustrative only and are not intendedto be limiting. Visible right image only 746 can be generated in avariety of ways. For example, for regions of visible right image only746 that contain only visible data, the visible data can be taken fromcaptured right image 738 and for regions that contain fluorescence, theregions are warped using the captured left image.

Again, note that while this description is necessarily linear anddescribes a single pass through the processing, the processes areoccurring in real-time and the various images are being continuouslyupdated to reflect the current state of tissue 703 as observed viaendoscope 701. Also, the various processes can proceed in parallel ifthe necessary information is available.

Single Stereoscopic Optical Path with a Camera Unit—AugmentedStereoscopic Display System 140D

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. For example, in a first mode, onlystereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, a fluorescence image issuperimposed on the visible images to create an augmented image, and thestereoscopic augmented image is output to the surgeon. In a third mode,a visible image for one eye of the stereoscopic display is blended withthe corresponding fluorescence image for that eye and only the visibleimage is presented for the other eye. Thus, the stereoscopic augmentedview has an augmented view for one eye and a normal view for the othereye in the stereoscopic display.

The video output may be toggled between these modes by using, e.g., afoot switch, a double click of the master grips that control thesurgical instruments, voice control, and other like switching methods.The toggle for switching between the various modes is represented inFIG. 7A as display mode select 760.

In response to a user input 820 (FIG. 8) the signal from display modeselect 760 (FIG. 7A) is provided to a display mode check operation 808in a user interface 793 that in turn provides a control signal to blendcircuit 751 and blend circuit 752. If the surgeon selects visible only,visible left image 736 and visible right image 746 are presented instereoscopic display 750 via stereoscopic display of visible imageprocess 811 (FIG. 8).

If the surgeon selects visible plus fluorescence in the right eye only,in blend process 812, blend circuit 751 passes visible left image 736 tostereoscopic display 750, while blend circuit 752 blends fluorescenceright image 744 and visible right image 746. Alternatively, blendcircuit 752 could pass visible and fluorescence right image 738 tostereoscopic display 750. The outputs of blend circuits 751, 752 arepresented in stereoscopic display 750 via stereoscopic display offluorescence image in one eye only and visible image process 813 (FIG.8) in generate stereoscopic video display of tissue process 205C.

If the surgeon selects visible plus fluorescence, in blend process 809,blend circuit 751 blends fluorescence left image 742 and visible leftimage 736, while blend circuit 752 blends fluorescence right image 744and visible right image 746.

Different image blending options, such as alpha blending, can beimplemented in blend circuits 751, 752. The outputs of blend circuits751, 752 are presented in stereoscopic display 750 via stereoscopicdisplay of visible and fluorescence images process 810 (FIG. 8)

The techniques previously described for enhancing the fluorescence imagein the stereoscopic display are also applicable to this embodiment.

Also, in the above described aspect, the left and right images could bereversed. Thus, the left image is an example of a first image and theright image is an example of a second image.

Time Division—Single Stereoscopic Optical Path with a Camera Unit

In still another aspect, the visible and fluorescence images arecaptured via the same stereoscopic optical path, but image capture istime division multiplexed. In this aspect, the same camera unit capturesdata for both the visible and fluorescence images, but at differenttimes. This time division is implemented by synchronizing a light sourceon/off with the video frame capture.

For example, in the embodiment of FIGS. 9A and 9B, a robotic surgicalsystem (not shown) includes a single stereoscopic optical path fortransporting light from tissue 903 to augmented stereoscopic visionsystem 900. Light from the single stereoscopic optical path is used togenerate a real-time stereoscopic video display of tissue 903 for thesurgeon operating the robotic surgical system. The stereoscopic videodisplay includes a three-dimensional view of tissue 903 blended with analternate image to highlight regions of interest in tissue 903 such asdiseased portions of tissue 903 and/or other tissue of interest, such asa nerve or organ.

In this example, a single endoscope 901 provides the stereoscopicoptical path from tissue 903 to hardware 920. Endoscope 901 has twolight channels making up the stereoscopic optical path and at least oneillumination channel for providing light to tissue 903. While it is notshown, endoscope 901 is held and moved by the robotic surgical system.See FIG. 1 for example.

In this example, augmented stereoscopic vision system 900 includes acombination light source 910, hardware 920, and a plurality ofcomputer-based methods 990. As shown in FIGS. 9A and 9B, a portion ofhardware 920 makes up image capture system 120E. Another portion ofhardware 920 and a portion of plurality of computer-based methods 990make up intelligent image processing system 130E. Yet another portion ofhardware 920 and another portion of plurality of computer-based methods990 make up augmented stereoscopic display system 140E. Within imagecapture system 120E and intelligent image processing system 130E, theportions that process visible images make up a visible imaging systemwhile the portions that process fluorescence images make up an alternateimaging system.

Also, method 1000 of FIG. 10A is implemented using augmentedstereoscopic vision system 900. As shown in FIG. 10A, method 1000includes a plurality of separate processes. Method 1000 is oneimplementation of method 200 (FIG. 2).

In one aspect, hardware 920 includes a single camera unit such as cameraunit 731 (FIG. 7B). Camera unit 731 includes a 3-chip charge-coupleddevice (CCD) sensor for each optical path of endoscope 901.

Hardware 920 also includes hardware circuits for performing thefunctions described more completely below. Plurality of computer-basedmethods 990 are, for example, software executing on a computerprocessor. In the following description, multiple hardware units aredescribed that perform the same function. This is for ease ofdescription only and is not intended to require the exact number shown.Depending on the implementation, a single instance of the hardware unitcould be used, or alternatively a number less than the number showncould be used so long as the hardware performs within a relevant timeperiod.

Time Division—Single Stereoscopic Optical Path with a CameraUnit—Illumination

Combination light source 910 (FIG. 9A) with fiber optic bundle 914 issimilar to any one of combination light sources 510A (FIG. 5C), 510B(FIG. 5D) and 510C (FIG. 5E) and the associated fiber optic bundles, aswell as the various aspects described above with respect to theimplementation of combination light source 310A (FIG. 3C). Accordingly,the description of those combination light sources is incorporatedherein by reference. However, combination light source 910 includes ameans for turning off and on at least one of the light sources.

As an example, combination light source 510A is selected as the startingpoint for combination light source 910 and a Pockels cell 911 isinserted in the laser light's path between turning mirror 516 andinjection mirror 513. Pockels cell 911 is connected to a laser/camerasync circuit 935. As explained more completely, in one aspect at a timet, Pockels cell 911 receives a signal from laser/camera sync circuit 935so that the laser beam passes through Pockels cell 911 and is injectedinto fiber optic cable 914 with the light from white light source 512A.Here, time t is associated with a frame, while time (t+1) is associatedwith a different frame.

At a time (t+1), Pockels cell 911 receives a signal from laser/camerasync circuit 935 so that the laser beam is blocked by Pockels cell 911and only the light from white light source 512A is injected into fiberoptic cable 914. Thus, for a first time interval, tissue 903 isilluminated with white light and with light that simulates fluorescencefrom tissue 903 and then for a second time interval, immediatelyfollowing the first time interval, tissue 903 is illuminated with onlythe white light. In this example, the laser beam is modulated on andoff. However, in view of the following description, system 900 could beimplemented with the white light source modulated on and off and withthe laser beam maintained continuously on.

Time Division—Single Stereoscopic Optical Path with a Camera Unit—ImageCapture System 120E

Laser/camera sync circuit 935 also provides a signal to camera sync 934and 933 in image capture system 120E. In response to that signal, camerasync 934 causes a frame to be captured in left CCD sensor 931A andcamera sync 933 causes the frame to be captured in right CCD sensor931B. Each CCD sensor is a 3-chip CCD sensor and so the captured imagehas red, green and blue color components. FIG. 9B is an example of thesynchronization between combination light source 910 and the framecapture.

For example, at time t, tissue 903 is illuminated with both the whitelight and the laser light and a signal Left Optical Path Capture, RightOptical Path Capture is sent to camera sync 934 and 933, respectively.Thus, at time t, a first stereoscopic frame 936A of visible left imageand fluorescence left image λL is captured in left CCD 931A. Also, attime t, a first stereoscopic frame 938A of visible right image andfluorescence right image λR is captured in right CCD 931B.

For example, at time (t+1), tissue 903 is illuminated with only thewhite light; the laser light is turned off; and a signal Left OpticalPath Capture, Right Optical Path Capture is sent to camera sync 934 and933, respectively. Thus, at time (t+1), a second stereoscopic frame 936Bof the visible left image is captured in left CCD 931A. Also, at time(t+1), a second stereoscopic frame 938A of the visible right image iscaptured in right CCD 931B.

As illustrated in FIG. 9C, for this example, the capture processcontinues and so the fluorescence image capture rate is one half thecapture rate of the visible image, e.g., visible data is collected inevery frame while fluorescence and visible data is collected in everyother frame. This capture rate is illustrative only and in view of thisdisclosure an appropriate capture rate for the fluorescence image can bechosen.

The left images from tissue 903 (FIG. 9A) are captured from a leftoptical channel of the stereoscopic optical path in endoscope 901, andthe right images from tissue 903 are captured from a right opticalchannel of the stereoscopic optical path in endoscope 901.

In the example of FIGS. 9A and 9B, two frames are shown as beingcaptured by the CCD sensor. This is for ease of illustration only and isnot intended to be limiting. As is known, prior to capture of the frameat time (t+1), the frame captured in the CCD sensor could be moved to abuffer, for example, for the processing described more completely below.

Time Division—Single Stereoscopic Optical Path with a CameraUnit—Intelligent Image Processing system 130E

Since the fluorescence and visible images are captured at differentframe rates, temporal registration 992 is used in synchronization of thefluorescence images with the visible images. In this example, spatialregistration is not needed. However, in one aspect, where spatialregistration is used, the spatial registration is done prior to temporalregistration 992. As described more completely below, the informationfrom temporal registration is used in applying a transformation togenerate missing fluorescence frames through image warping as well as ingenerating individual images when the visible and fluorescence imagesare captured together.

Thus, in this example, temporal image registration 992 receives asinputs, each of captured frames 936A, 936B, 938A and 938B. Temporalimage registration 992 also receives an input from capture mode select945. In this example, three capture modes are considered. A firstcapture mode is, as described above, a time division mode 945B wherevisible plus fluorescence images and visible images only are captured.In a second capture mode, continuous visual mode 945A, only visibleimages are captured and the fluorescence excitation light source is heldoff. In the third capture mode, referred to as an extended mode, onlyvisible images are captured because the fluorescence images are nolonger available and so the fluorescence left and right images aresynthesized, as described more completely below. Note that while in thesecond and third capture modes, the setup for the modes is different,the capture, processing and display processes are effectivelyequivalent.

Using the temporal image registration information of visible left image936B at time (t+1) with captured visible left image combined withfluorescence left image 936A at time t, image warper 940A generates acombined visible left image and fluorescence left image for time (t+1).Image warper 940A compensates for any motion between times t and (t+1).

The generated combined visible left image and fluorescence left imagefor time (t+1) is supplied to image subtractor 942A as a first input.Image subtractor 942A receives visible left image 936B at time (t+1)from left CCD 931A as a second input. Image subtractor 942A subtractsvisible left image 936B from the generated combined visible left imageand fluorescence left image for time (t+1) to generate artificialfluorescence left image 947A at time (t+1).

The generated fluorescence left image 947A at time (t+1) is an input toimage warper 940B. Image warper 940B also receives temporal imageregistration information as an input. Image warper 940A generates afluorescence left image 947B for time t from generated fluorescence leftimage 947A at time (t+1). Image warper 940B compensates for any motionbetween times t and (t+1).

Generated fluorescence left image 947B for time t is supplied to imagesubtractor 942B as a first input. Image subtractor 942A receivescaptured visible left image combined with fluorescence left image 936Aat time t from left CCD 931A as a second input. Image subtractor 942Bsubtracts generated fluorescence left image 947B for time t fromcaptured visible left image combined with fluorescence left image 936Aat time t to generate visible left image 944L at time t.

Using the temporal image registration information of visible right image938B at time (t+1) with captured visible right image combined withfluorescence right image 938A at time t, image warper 941A (FIG. 9B)generates a combined visible right image and fluorescence right imagefor time (t+1). Image warper 941A compensates for any motion betweentimes t and (t+1).

The generated combined visible right image and fluorescence right imagefor time (t+1) is supplied to image subtractor 943A as a first input.Image subtractor 932A receives visible right image 938B at time (t+1)from right CCD 931B as a second input. Image subtractor 943A subtractsvisible right image 938B from the generated combined visible right imageand fluorescence right image for time (t+1) to generate artificialfluorescence right image 946A at time (t+1).

The generated artificial fluorescence right image 946A at time (t+1) isan input to image warper 941B. Image warper 941B also receives temporalimage registration information as an input. Image warper 941B generatesa fluorescence right image 946B for time t from generated artificialfluorescence right image 946A at time (t+1). Image warper 941Bcompensates for any motion between times t and (t+1).

Generated fluorescence right image 946B for time t is supplied to imagesubtractor 943B as a first input. Image subtractor 943A receivescaptured visible right image combined with fluorescence right image 938Aat time t from right CCD 931B as a second input. Image subtractor 943Bsubtracts generated fluorescence right image 946B for time t fromcaptured visible right image combined with fluorescence right image 938Aat time t to generate visible right image 944R at time t.

Fluorescence left image 947B and fluorescence right image 946B are astereoscopic pair of fluorescence images. Similarly, artificialfluorescence left image 947A and artificial fluorescence right image946A are a stereoscopic pair of fluorescence images.

In FIGS. 9A and 9B, the fluorescence and visible data for the frames attimes t and (t+1) are shown as both being supplied to blend circuits 951and 952. This is for ease of understanding only. The two frames would beprovided in the proper sequence so that the stereoscopic displaypresented to the surgeon flows in the proper time sequence even thoughthe video sequence may be delayed by one or more frames to allow for theprocessing described above.

When capture mode select 945 (FIG. 9A) is in continuous visual mode945A, laser module 517 is turned off and so only visible images arecaptured. In this mode, a fluorescence image is not generated ordisplayed, and so the captured visible images are simply displayed onstereoscopic display 950 in the normal way.

Augmented system 900, in this example, also includes the ability toprovide a fluorescence image indicating tissue of interest for a longperiod of time, even after an agent in the tissue no long fluoresces. Inthis situation irrespective of the configuration of combination lightsource 910, only visible images are captured and registered in temporalregistration 992. This mode is referred as the extended mode.

In the extended mode, temporal registration 992 provides the imageregistration information to synthesize fluorescence left image 949 andto synthesize fluorescence right image 948 in the extended mode.Synthesize fluorescence left image 949 also receives as input the lastfluorescence left image generated, i.e., fluorescence left image 947A.Synthesize fluorescence left image 949 generates a synthesizedfluorescence left image by using the temporal registration informationto move fluorescence left image 947A into the correct position withrespect to the current visible only left image. In one aspect, theprocess used to generate a synthetic fluorescence left image isequivalent to that just described to generate an artificial fluorescenceleft image.

Similarly, synthesize fluorescence right image 948 also receives asinput the last fluorescence right image generated, i.e., fluorescenceright image 946A. Synthesize fluorescence right image 948 generates asynthesized fluorescence right image by using the registrationinformation to move fluorescence right image 946 into the correctposition with respect to the current visible only right image. In oneaspect, the process used to generate a synthetic fluorescence rightimage is equivalent to that just described to generate an artificialfluorescence right image.

Again, note that while this description is necessarily linear anddescribes a single pass through the processing, the processes arerepeating in real-time and the various images are being continuouslyupdated to reflect the current state of tissue 903 as observed viaendoscopes 901 using the processes just described.

Time Division—Single Stereoscopic Optical Path with a CameraUnit—Augmented Stereoscopic Display System 140E

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. For example, in a first mode, onlystereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, a fluorescence image issuperimposed on the visible images to create an augmented image, and thestereoscopic augmented image is output to the surgeon.

The video output may be toggled between these modes by using, e.g., afoot switch, a double click of the master grips that control thesurgical instruments, voice control, and other like switching methods.The toggle for switching between the two modes is represented in FIG. 9Aas display mode select 960 which generates a signal that is provided touser interface 993 that in turn outputs a signal to blend circuits 951,952, which function in the same way as previously described for theother blend circuits.

The techniques previously described for enhancing the fluorescence imagein the stereoscopic display are also applicable to this embodiment.

In addition, in some aspects, another display mode that generates anaugmented display with fluorescence and visible images may beimplemented. This display mode can be a variation of the first displaymode or a completely different display mode. In either case, thefluorescence image flickers on and off in the stereoscopic display andso is not continuously displayed. This feature can be implemented, forexample, using intelligent image processing system 130E to supply theappropriate frames to system 140E so that the desired flicker rate isobtained. Alternatively, the laser light from combination light source910 can be modulated to provide the flicker in the display. The flickermode can be used in the other aspects described herein and so is notrepeated for each one.

The above description of time division works for a first frame rate forthe visible images, e.g., 30 frames per second, and a second frame ratefor capture of fluorescence images. This description can be combinedwith the other aspects described above to provide time division in anyof the aspects. Also, for time division multiplexing aspects, or inother image capture and processing aspects, filtering may be done toremove artifacts including motion artifacts and/or illuminationartifacts if the endoscope or tissue is moved or the illuminationchanges.

The aspects in FIGS. 9A and 9B are illustrative only and in view of thedisclosure those knowledgeable in the field can implement a variety ofconfigurations to achieve similar results. For example, FIG. 9Dillustrates one alternative embodiment. In this example, system 900Aincludes hardware 920A that includes an alternative intelligent imageprocessing system 130E. Here, Current Frame selects the appropriatestored frame to send to augmented stereoscopic display system 140E. Forexample, frames 936A and 936B are accessed as a stereoscopic pair andsent to system 140E. Next, frame 936B and artificial fluorescence leftimage 947A and frame 938B and artificial fluorescence right image 946Aare provided to system 140E. The other elements in FIG. 9D work in thesame way as described above for elements with the same reference numeralin FIGS. 9A and 9B.

FIG. 9E illustrates another aspect of intelligent image processingsystem 130E that could be incorporated for example in the system ofFIGS. 9A and 9B. Here, only the processing of the images captured fromthe right optical channel in the stereoscopic optical path is shown. Thehardware and processing for the images captures from the left opticalchannel is equivalent and so is not shown.

In this aspect, image warper 941B receives the visible right imagecaptured at time (t−1) and the appropriate registration information.Image warper 941B generates visible right image 944R at time t. Imagesubtractor 943B receives as inputs visible right image 944R at time tand captured visible plus fluorescence right image 938A at time t. Imagesubtractor 943B subtracts visible right image 944R from captured visibleplus fluorescence right image 938A to generate artificial fluorescenceright image 946B at time t.

FIG. 10A is a process flow diagram for general time divisionmultiplexing in which a combination visible fluorescence image frame iscaptured and then N, where N is an integer, visible only image framesare captured. For convenience, FIG. 10A is explained using the apparatusof FIG. 9A. FIG. 10A assumes that any initialization has been completedand process 1000 is in operation.

In capture visible and fluorescence images process 1001, image capturesystem 120E captures a stereoscopic pair of combined visible andfluorescence images 936A, 936B. Blended visible and fluorescence imagesare displayed on stereoscopic display 950 in stereoscopic display ofimages process 1002.

Initialize frame counter process 1003 initializes a frame counter to N(In the example described above N is one), and then in turn-offfluorescence excitation process 1004, laser/camera sync circuit 935causes Pockels cell 911 to block the laser light beam.

Next, laser/camera sync circuit 935 causes visible images 936B, 938B tobe captured in capture visible images process 1005. In registrationprocess 1006, visible left image 936B is registered to capturedcombination left image 936A and right visible image 938B is registeredto captured combination right image 938A.

Generate fluorescence left and right images process 1007 usesintelligent image processing system 130E to generate these images asdescribed above with respect to FIGS. 9A and 9B. The visible andfluorescence images associated with the next frame in the time sequencefor display are blended and displayed on stereoscopic display 950 inprocess 1008.

The frame counter is decremented in process 1009 and counter equal tozero check operation 1010 determines whether to capture another set ofvisible images or another set of combination visible and fluorescenceimages. If the frame counter is not equal to zero, processes 1005 to1009 are repeated. If the frame counter is equal to zero, process 1011turns on the fluorescence excitation source and transfers to process1001.

FIG. 10B illustrates the visible and fluorescence frame streams inprocess 1000. As described above and shown in FIG. 10B, at time t,visible and fluorescence frames are captured.

From time (t+1) to time (t+N), only visible frames are captured. Duringthis time interval, the fluorescence frame captured at time t istemporally registered to each captured visible frame and then thefluorescence frame captured at time t is warped to produce theartificial fluorescence frame for the corresponding time as shown inFIG. 10B. In one aspect, the captured and artificial fluorescence framescould be integrated and the integrated result displayed. Notice that theartificial fluorescence frames are used to synchronize the frame rate ofthe fluorescence image with the frame rate of the visible image in thestereoscopic display.

In more general terms, the fluorescence image may be artificiallysustained in the video output to the surgeon between the frames thatcapture the fluorescence image. The artificially sustained fluorescenceimage is blended with the visible images as the camera switches betweencapturing visible and fluorescence images.

Also, as another example and as indicated above, the fluorescence imagemay be sustained after an injected agent no longer fluoresces so thatthe fluorescing region is still visible to the surgeon. In one aspect,the sustained image output is automatically stopped if the camera ismoved so that the surgeon does not see a false blending of fluorescenceand visible images. If the fluorescing region in the fluorescence imageis spatially registered to the visible image, however, the sustainedfluorescence image may be output because it is correctly blended withthe visible image. As discussed above, artifacts may be filtered fromthe output display.

Time Division—Single Stereoscopic Optical Path with a Camera Unit thatCaptures Fluorescence Image Combined with a Visible Color Component

In still another aspect, the visible and fluorescence images arecaptured via the same stereoscopic optical path, but image capture istime division multiplexed and the fluorescence image is captured withone of the visible color components, e.g., the red color component. Inthis aspect, the same camera unit captures data for both the visible andfluorescence images, but at different times. This time division isimplemented by synchronizing a light source on/off with the video framecapture.

For example, in the embodiment of FIG. 11A, a robotic surgical system(not shown) includes a single stereoscopic optical path for transportinglight from tissue 1103 to augmented stereoscopic vision system 1100.Light from the single stereoscopic optical path is used to generate areal-time stereoscopic video display of tissue 1103 for the surgeonoperating the robotic surgical system. The stereoscopic video displayincludes a three-dimensional view, sometimes called presentation, oftissue 1103 blended with an alternate image to highlight regions ofinterest in tissue 1103 such as diseased portions of tissue 1103 and/orother tissue of interest, such as a nerve or organ.

In this example, a single endoscope 1101 provides the stereoscopicoptical path from tissue 1103 to hardware 1120. Endoscope 1101 has twolight channels making up the stereoscopic optical path and at least oneillumination channel for providing light to tissue 1103. While it is notshown, endoscope 1101 is held and moved by the robotic surgical system.See FIG. 1 for example.

In this example, augmented stereoscopic vision system 1100 includes acombination light source 1110, hardware 1120, and at least onecomputer-based method 1190. As shown in FIG. 11A, a portion of hardware1120 makes up image capture system 120F. Another portion of hardware1120 makes up intelligent image processing system 130F. Yet anotherportion of hardware 1120 and a computer based method make up augmentedstereoscopic display system 140F. Within image capture system 120F andintelligent image processing system 130F, the portions that processvisible images make up a visible imaging system while the portions thatprocess fluorescence images make up an alternate imaging system.

Also, method 1200 of FIG. 12 is implemented using augmented stereoscopicvision system 1100. As shown in FIG. 12, method 1200 includes aplurality of separate processes.

In one aspect, hardware 1120 includes a single camera unit such ascamera unit 731 (FIG. 7B). Camera unit 731 includes a 3-chipcharge-coupled device (CCD) sensor for each optical path of endoscope1101.

Hardware 1120 also includes hardware circuits for performing thefunctions described more completely below. At least one computer-basedmethod 1190 is, for example, software executing on a computer processor.

Time Division—Single Stereoscopic Optical Path with a Camera Unit thatCaptures Fluorescence Image Combined with a Visible ColorComponent—Illumination

Combination light source 1110 with fiber optic bundle 1114 is similar toany one of combination light sources 510A (FIG. 5C), 510B (FIG. 5D) and510C (FIG. 5E) and the associated fiber optic bundles, as well as thevarious aspects described above with respect to the implementation ofcombination light source 310A (FIG. 3C). Accordingly, the description ofthose combination light sources is incorporated herein by reference.However, combination light source 1110 includes a means for turning offand on at least one of the light sources and so light source 910 isused, as and example, and the above description of light source 910A isincorporated herein by reference.

In one aspect, at a time t, Pockels cell 911 receives a signal fromlaser/camera sync circuit 1135 so that the laser beam is blocked byPockels cell 911 and only the light from white light source 512A isinjected into fiber optic cable 1114. At a time (t+1), Pockels cell 911receives a signal from laser/camera sync circuit 1135 so that the laserbeam passes through Pockels cell 911 and is injected into fiber opticcable 1114 with the light from white light source 512A. Here, time t isassociated with a frame, while time (t+1) is associated with a differentframe.

Thus, for a first time interval, tissue 1103 is illuminated with onlythe white light and then for a second time interval immediatelyfollowing the first time interval, tissue 1103 is illuminated with whitelight and with light that simulates fluorescence from tissue 1103. Inthis example, the laser beam is modulated on and off. However, in viewof the following description, system 1100 could be implemented with thewhite light source modulated on and off and with the laser beammaintained continuously on.

Time Division—Single Stereoscopic Optical Path with a Camera Unit thatCaptures Fluorescence Image Combined with a Visible ColorComponent—Image Capture System 120F

Laser/camera sync circuit 1135 also provides a signal to camera sync1134 and 1133. In response to that signal, camera sync 1134 causes aframe to be captured in left CCD sensor 1131A and camera sync 1133causes a frame to be captured in right CCD sensor 1131B. Each CCD sensoris a 3-chip CCD sensor. FIG. 11B is an example of the synchronizationbetween combination light source 1110 and the frame capture.

For example, at time t, tissue 1103 is illuminated with the white lightand a signal Left Optical Path Capture, Right Optical Path Capture issent to camera sync 1134 and 1133, respectively. Thus, at time t, afirst stereoscopic frame 1136A of visible red, green and blue componentsof a visible left image is captured in left CCD 1131A. Also, at time t,a first stereoscopic frame 1138A of visible red, green and bluecomponents of a visible right image is captured in right CCD 1131B.

For example, at time (t+1), tissue 1103 is illuminated with both thewhite light and the laser light, and a signal Left Optical Path Capture,Right Optical Path Capture is sent to camera sync 1134 and 1133,respectively. Thus, at time (t+1), a second stereoscopic frame 1136B ofvisible red, green and blue components of a visible left image iscaptured in left CCD 1131A. However, the visible red component iscombined with the fluorescence left image λL so that the combined imageis captured in left CCD 1131A. Also, at time (t+1), a secondstereoscopic frame 1138A of visible red, green and blue components of avisible right image is captured in right CCD 1131B. However, the visiblered component is combined with the fluorescence right image λR so thatthe combined image is captured in right CCD 1131B. This capture rate isillustrative only and in view of this disclosure an appropriate capturerate for the fluorescence image can be chosen.

The left images from tissue 1103 (FIG. 11A) are captured from a leftoptical channel of the stereoscopic optical path in endoscope 1101, andthe right images from tissue 1103 are captured from a right opticalchannel of the stereoscopic optical path in endoscope 1101.

In the example of FIG. 11A, two frames are shown as being captured bythe CCD sensor. This is for ease of illustration only and is notintended to be limiting. As is known, prior to capture of the frame attime (t+1), the frame captured in the CCD sensor could be moved to abuffer, for example, for the processing described more completely below.

Time Division—Single Stereoscopic Optical Path with a Camera Unit thatCaptures Fluorescence Image Combined with a Visible ColorComponent—Intelligent Image Processing System 130E

Since the fluorescence image is captured with one of visible colorcomponents, it necessary to extract the fluorescence image so that thefluorescence image can be processed to highlight the fluorescence tissuein the stereoscopic visual display. The processing of the left and rightimages is similar and so in the following description only the leftchannel is considered.

Herein, the red, green and blue visible components captured at time tare represented by R_(t), G_(t). and B_(t), respectively. The componentcaptured at time (t+1) are represented by (R+λ)_(t+1), G_(t+1), andB_(t+1).

Fluorescence image and artifacts generator 1140 (FIG. 11A), in oneaspect, uses a frame-to-frame subtraction process 1201 of process 1200(FIG. 12) to generate a fluorescence image with possible artifactsintroduced by illumination changes as well as motion artifacts.Frame-to-frame subtraction process 1201 subtracts the frame captured attime t from the frame captured at time (t+1). For example,

I _(R)=(R+λ)_(t+1) −R _(t)

I _(G) =G _(t+1) −G _(t)

I _(B) =B _(t+1) −B _(t)

where I_(R), I_(G), and I_(B) are frame-to-frame color componentdifferences for the red, green and blue color components, respectively.More specifically, frame-to-frame red color component difference I_(R)is the fluorescence image combined with possible artifacts of the redcolor component; and frame-to-frame green color component differenceI_(G) and frame-to-frame blue color component difference I_(B) are thepossible artifacts of the green and blue color components, respectively.The phrase “possible artifacts” is used because tissue 1103, endoscope1101 and instruments may not move between the two frames and thelighting may be stable in such a case there would be no artifacts.

To separate artifacts from the fluorescence image, scale system 1141(FIG. 11A), in one aspect, implements normalize process 1202 (FIG. 12).Scale system 1141 is optional and is not used in all aspects. However,in this aspect, normalize process 1202 processes frame-to-frame colorcomponent differences I_(R), I_(G), and I_(B) so that the differenceshave a common scale. For example, in one aspect, the mean for each colorcomponent difference is subtracted from the color component differenceand the result is scaled to a unit variance. For example,

${\hat{I}}_{R} = \frac{I_{R} - {\overset{\_}{I}}_{R}}{\sigma_{I_{R}}^{2}}$${\hat{I}}_{G} = \frac{I_{G} - {\overset{\_}{I}}_{G}}{\sigma_{I_{G}}^{2}}$${\hat{I}}_{B} = \frac{I_{B} - {\overset{\_}{I}}_{B}}{\sigma_{I_{B}}^{2}}$

where I_(R), I_(G) and I_(B) and I_(R), I_(G) and I_(B), respectively,are the same elements. A bar over the color component differencerepresents the mean for that color components and a square of σrepresents the variance. The determination of the mean and variancecould be based on the whole frame or alternatively a smaller region ofthe frame.

If there was motion or an illumination change between the times of thecapture of the two frames, the artifacts in the three color componentsbetween the two frames are usually similar, but may not be exactlyidentical. Thus, the artifacts of the normalized green and blue colorcomponents between the two frames can be used to approximate theartifacts in the normalized red component. Thus, in one aspect, thenormalized blue and green components are used to ameliorate the effectsof artifacts between the two frames in the fluorescence image.

Thus, fluorescence image extractor 1142 (FIG. 11A), in one aspectimplements cross-channel subtraction process 1203 (FIG. 12).Cross-channel subtraction process 1203 subtracts the normalized blue andgreen frame-to-frame color component differences from the normalizedframe-to-frame red color component that includes fluorescence image λ toobtain the fluorescence image F. Specifically, in one aspect,cross-channel subtraction process 1203 generates fluorescence image Fas:

F=|Î _(R) −Î _(B) |+|Î _(R) −Î _(G)|

Fluorescence image enhancement 1143 (FIG. 11A), in one aspect,implements enhance fluorescence image process 1204 (FIG. 12). Enhancefluorescence image process 1204 optionally scales fluorescence image Fand changes the color from red to either green or blue, e.g., falsecolors fluorescence image F so that when fluorescence left image 1144 isblended with a visible image, the fluorescence image stands out.Artificial fluorescence left image 1144 and artificial fluorescenceright image 1149 are a stereoscopic pair of fluorescence images.

In one aspect, optional sparse tissue tracking is included. For example,temporal registration for pixels declared as motion regions is used todetermine whether there truly was motion in each of the motion regions.In addition, image filtering and thresholding can be used to clean upthe final fluorescence image.

The above aspects of process 1200 are illustrative only and are notintended to be limiting. Various variants of these processes can beimplemented in the hardware. For example,

Frame-frame subtraction

I _(t+1) −I _(t) ={R _(t+1) −R _(t)+Δ_(t+1) ,G _(t+1) −G _(t) ,B _(t+1)−B _(t)}

Detect the motion/illumination change regions (MIR)

if (G _(t+1) −G _(t))+(B _(t+1) −B _(t))>threshold_1  MIR:

Determine fluorescence region (FR) after post processing if

if NOT MIR & abs((R _(t+1) −R _(t))+Δ_(t+1)−(G _(t+1) −G_(t)))>threshold_2  FR:

In one aspect the thresholds are empirically determined.

Elements 1145 to 1149 function in a matter to that just described forthe left channel and also implement process 1200. Accordingly, theimplementation of elements 1145 to 1149 follows directly from the abovedescription.

This processing continues for each pair of captured frames. Variousalternatives can be used to generate artificial images to provide theframe rate needed for display of the visible and fluorescence images.For example, the images can simply be repeated. Alternatively, oncemultiple frames have been processed interpolation could be used togenerate artificial frames for the fluorescence image and/or visibleimage as needed.

Time Division—Single Stereoscopic Optical Path with a Camera Unit thatCaptures Fluorescence Image Combined with a Visible ColorComponent—Augmented Stereoscopic Display System 140F

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. For example, in a first mode, onlystereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, a fluorescence image issuperimposed on the visible images to create an augmented image, and thestereoscopic augmented image is output to the surgeon.

The video output may be toggled between these modes by using, e.g., afoot switch, a double click of the master grips that control thesurgical instruments, voice control, and other like switching methods.The toggle for switching between the two modes is represented in FIG.11A as display mode select 1160 which generates a signal that isprovided to blend circuits 1151, 1152, which function in the same way aspreviously described for the other blend circuits.

The techniques previously described for enhancing the fluorescence imagein the stereoscopic display are also applicable to this embodiment.

Single Stereoscopic Optical Path with a Modified Camera Unit

In still another aspect, the visible and fluorescence images arecaptured via the same stereoscopic optical path, but the fluorescenceimage is again captured with one of the visible color components. Inthis aspect, the same camera unit captures data for both the visible andfluorescence stereoscopic images but a prism, as explained morecompletely below, combines the fluorescence image with one of thevisible color components.

For example, in the embodiment of FIG. 13A, a robotic surgical system(not shown) includes a single stereoscopic optical path for transportinglight from tissue 1303 to augmented stereoscopic vision system 1300.Light from the single stereoscopic optical path is used to generate areal-time stereoscopic video display of tissue 1303 for the surgeonoperating the robotic surgical system. The stereoscopic video displayincludes a three-dimensional view of tissue 1303 with an alternate imageto highlight regions of interest in tissue 1303 such as diseasedportions of tissue 1303 and/or other tissue of interest, such as a nerveor organ.

In this example, a single endoscope 1301 provides the stereoscopicoptical path from tissue 1303 to hardware 1320. Endoscope 1301 has twolight channels making up the stereoscopic optical path and at least oneillumination channel for providing light to tissue 1303. While it is notshown, endoscope 1301 is held and moved by the robotic surgical system.See FIG. 1 for example.

In this example, augmented stereoscopic vision system 1300 includescombination light source 1310, hardware 1320, and at least onecomputer-based method 1390. As shown in FIG. 13A, a portion of hardware1320 makes up image capture system 120G. In this aspect, an intelligentimage processing system is not used. Yet another portion of hardware1320 and at least one computer-based method 1393 make up augmentedstereoscopic display system 140G. Within image capture system 120G, theportions that process visible images make up a visible imaging systemwhile the portions that process fluorescence images make up an alternateimaging system.

In one aspect, hardware 1320 includes a single camera unit such as amodified camera unit 731 (FIG. 7B). Camera unit 731 includes a 3-chipcharge-coupled device (CCD) sensor for each optical path of endoscope1301.

Hardware 1320 also includes hardware circuits for performing thefunctions described more completely below. Computer-based methods 1390are, for example, software executing on a computer processor.

Single Stereoscopic Optical Path with a Modified CameraUnit—Illumination

Combination light source 1310 with fiber optic bundle 1314 is similar tocombination light source 910 and fiber optic bundle 914. Accordingly,the description of that combination light source is incorporated hereinby reference. However, in combination light source 1310 the control of ameans for turning off and on at least one of the light sources isdifferent from that in combination light source 910.

In this aspect, Pockels cell 911 receives a control signal from lightcontroller 1335 that in turn receives a signal from user interface 1393.When the surgeon selects visible only, a signal is applied to Pockelscell 911 so that the laser beam is blocked by Pockels cell 911 and onlythe light from white light source 512A is injected into fiber opticcable 1314. When the surgeon selects visible plus fluorescence, bothlight sources provide a beam that is injected into fiber optic cable1314.

Single Stereoscopic Optical Path with a Modified Camera Unit—ImageCapture System 120G

In this aspect, the camera unit is modified so that the light from eachoptical path is passed through a modified prism 1334, 1333. Each ofmodified prisms 1334, 1333, for example, has the characteristics shownin FIG. 13B.

Modified prisms 1334, 1333 split the visible plus fluorescence imagesfrom the optical paths into typical RGB components 1381, 1382, 1383(Note the color of the component is represented by the characteristic ofthe line—blue by a dashed line; green by a solid line; and red by adotted line.) However, in this example, modified prisms 1334, 1333generate not only a blue color component 1381 but also a second bluepeak 1384 that is in near infrared region. Thus, when the fluorescenceimage is in the near infrared, this prism separates the combined visibleand fluorescence images into a visible RGB image and a blue fluorescenceimage in the near infrared. Modified prism 1334, 1333 are made in aconventional fashion, except a portion of the prism that normally passesonly one visible color component is modified to pass both that visiblecolor component and another component that is separated and removed fromthe color component. The another component corresponds to a fluorescenceimage.

While in this aspect, dual peaks are provided with respect to the bluecomponent, depending upon the fluorescence wavelength, the prism can bemodified to obtain the desired results for any of the color components.In this aspect, the CCD for the blue color component accumulates bothpeak 1481 and peak 1484 and so captures the visible blue image from theoptical path combined with the fluorescence image from that opticalpath.

Thus, as illustrated in FIG. 13A, when both the beam from light source512A and the beam from light source 517 illuminate tissue 1303, a redleft CCD in left CCD 1331A captures a visible red left image 1336A; agreen left CCD in left CCD 1331A captures a visible green left image1336B; and a blue left CCD in left CCD 1331A captures a visible blueleft image combined with a fluorescence left image 1336C. Similarly, ared right CCD in left CCD 1331B captures a visible red right image1338A; a green right CCD in right CCD 1331B captures a visible greenright image 1338B; and a blue right CCD in right CCD 1331B captures avisible blue right image combined with a fluorescence right image 1338C.

When the laser beam is not injected in fiber optic bundle 1314 and onlylight from light source 512A is injected, the red left CCD in left CCD1331A captures a visible red left image 1336A; the green left CCD inleft CCD 1331A captures a visible green left image 1336B; and the blueleft CCD in left CCD 1331A captures only a visible blue left image1336C. This is why “With Fluorescence Left Image” is enclosed inparentheses in FIG. 13A, because the fluorescence left image is notalways captured. Similarly, the red right red CCD in right CCD 1331Bcaptures a visible red right image 1338A; the green right CCD in rightCCD 1331B captures a visible green right image 1338B; and the blue rightCCD in right CCD 1331B captures only a visible blue right image 1338C.This is why “With Fluorescence Right Image” also is enclosed inparentheses in FIG. 13A, because the fluorescence right image is notalways captured.

Single Stereoscopic Optical Path with a Modified Camera Unit—AugmentedStereoscopic Display System 140G

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. For example, in a first mode, only real-timestereoscopic visible images are output to the surgeon, as in the daVinci® Surgical System. In a second mode, a fluorescence image issuperimposed on the visible image to create an augmented image, and thereal-time stereoscopic augmented image is output to the surgeon.

As indicated above, the video output may be toggled between these modesby using, e.g., a foot switch, a double click of the master grips thatcontrol the surgical instruments, voice control, and other likeswitching methods. The toggle for switching between the two modes isrepresented in FIG. 13A as display mode select 1360 which generates asignal that is provided to user interface 1393 that in turn provides acontrol signal to light controller 1335 as previously described.

The techniques previously described for enhancing the fluorescence imagein the stereoscopic display are also applicable to this embodiment. Whensuch techniques are used an intelligent image processing system would beincluded in system 1300.

In addition, in some aspects, another display mode that generates anaugmented display with fluorescence and visible images may beimplemented. This display mode can be a variation of the first displaymode or a completely different display mode. In either case, thefluorescence image flickers on and off in the stereoscopic display andso is not continuously displayed. This feature can be implemented, forexample, using an intelligent image processing system to supply theappropriate frames to system 140G so that the desired flicker rate isobtained. Alternatively, the laser light from combination light source1310 can be modulated to provide the flicker in the display.

In one aspect, method 1400 (FIG. 14) is implemented using augmentedstereoscopic vision system 1300. In illuminate tissue process 201E,light from combination light source 1310 illuminates tissue 1303.

Light from tissue 1303 is split into the various components, asdescribed above, in pre-process light from tissue process 202E. Thevarious components including the components combined with thefluorescence images are captured in capture stereoscopic visible andstereoscopic fluorescence images process 203E.

Based upon the user input to user input process 1420, a display modecheck operation 1408 in user interface 1393 configures combination lightsource, if necessary, and performs one of stereoscopic display ofvisible and fluorescence images process 1410 and stereoscopic display ofvisible image only process 1411. Process 1410 and 1411, in generatestereoscopic video display of tissue process 205E, generate the displaysthat were described above with respect to augmented stereoscopic visionsystem 1300.

Time-Division—Single Stereoscopic Optical Path with a Single CCD CameraUnit

In still another aspect, the visible and fluorescence images arecaptured via the same stereoscopic optical path, but image capture istime division multiplexed. In this aspect, the same camera unit capturesdata for both the color components of the visible image and thefluorescence image, but at different times. This time division isimplemented by synchronizing capture with filtering using a rotatingfilter, as described more completely below.

For example, in the embodiment of FIG. 15, a robotic surgical system(not shown) includes a single stereoscopic optical path for transportinglight from tissue 1503 to augmented stereoscopic vision system 1500.Light from the single stereoscopic optical path is used to generate areal-time stereoscopic video display of tissue 1503 for the surgeonoperating the robotic surgical system. The stereoscopic video displayincludes a three-dimensional view of tissue 1503 blended with analternate image to highlight regions of interest in tissue 1503 such asdiseased portions of tissue 1503 and/or other tissue of interest, suchas a nerve or organ.

In this example, augmented stereoscopic vision system 1500 includescombination light source 1510, hardware 1520, and at least onecomputer-based method 1590. As shown in FIG. 15, a portion of hardware1520 makes up image capture system 120H. Another portion of hardware1520 makes up intelligent image processing system 130H. Yet anotherportion of hardware 1520 and a user interface 1593 in computer-basedmethods 1590 make up augmented stereoscopic display system 140H. Withinimage capture system 120H and intelligent image processing system 130H,the portions that process visible images make up a visible imagingsystem while the portions that process fluorescence images make up analternate imaging system.

In one aspect, hardware 1520 includes a single camera unit such as amodified camera unit 731 (FIG. 7B). Camera unit 731 includes a singlecoupled device (CCD) sensor for each optical path of endoscope 1501.

Hardware 1520 also includes hardware circuits for performing thefunctions described more completely below. Computer-based methods 1590are, for example, software executing on a computer processor.

Time-Division—Single Stereoscopic Optical Path with a Single CCD CameraUnit—Illumination

Combination light source 1510 with fiber optic bundle 1514 is similar toany one of combination light sources 510A (FIG. 5C), 510B (FIG. 5D) and510C (FIG. 5E) and the associated fiber optic bundles, as well as thevarious aspects described above with respect to the implementation ofcombination light source 310A (FIG. 3C). Accordingly, the description ofthose combination light sources is incorporated herein by reference.

Time-Division—Single Stereoscopic Optical Path with a Single CCD CameraUnit—Image Capture System 120G

In this aspect, tissue 1503 is illuminated with both the white light andthe laser light from combination light source 1510. The captureoperations for the left and right images are equivalent.

Rotating filter 1532A includes four band pass filters: a visible redfilter, a visible green filter, a visible blue filter and a fluorescencefilter. Rotating filter 1532B is similarly configured. Rotating filters1532A, 1532B are coupled with the capture of an image in the single CCDof the camera unit, e.g., left CCD 1531A for the filtered light from theleft optical path of endoscope 1501 and right CCD 1533A for filteredlight from the right optical path of endoscopic 1501.

At time t, rotating filter 1532A filters light from the left opticalpath of endoscope 1501 with the red filter and so left CCD 1531Acaptures red left image 1534L at time t. At time t+1, rotating filter1532A filters light from the left optical path of endoscope 1501 withthe green filter and so left CCD 1531A captures green left image 1535Lat time t+1. At time t+2, rotating filter 1532A filters light from theleft optical path of endoscope 1501 with the blue filter and so left CCD1531A captures blue left image 1536L at time t+2. At time t+3, rotatingfilter 1532A filters light from the left optical path of endoscope 1501with the fluorescence filter and so left CCD 1531A captures fluorescenceleft image 1537L at time t+3.

At time t, rotating filter 1532B filters light from the right opticalpath of endoscope 1501 with the red filter and so right CCD 1533Acaptures red right image 1534R at time t. At time t+1, rotating filter1533A filters light from the right optical path of endoscope 1501 withthe green filter and so right CCD 1533A captures green right image 1535Rat time t+1. At time t+2, rotating filter 1532B filters light from theright optical path of endoscope 1501 with the blue filter and so rightCCD 1533A captures blue right image 1536R at time t+2. At time t+3,rotating filter 1532B filters light from the right optical path ofendoscope 1501 with the fluorescence filter and so right CCD 1533Acaptures fluorescence right image 1537R at time t+3.

The capture process starts over at time t+4 in this aspect. In theexample of FIG. 15, four frames are shown as being captured by the CCDsensor. This is for ease of illustration only and is not intended to belimiting. As is known, prior to capture of the frame, the previouslycaptured frame in the CCD sensor could be moved to a buffer, forexample, for the processing described more completely below.

In this example, the two light sources in combination light source weremaintained continuously on. In another aspect, the white light source isheld on for capture at times t, t+1, and t+2 and then turned off beforethe capture at time t+3. For the capture at time t+3, the laser isturned on. In this aspect, the fluorescence filter would not be used.

Time Division—Single Stereoscopic Optical Path with a Single CCD CameraUnit—Intelligent Image Processing System 130H

Since the fluorescence and visible components images are captured atdifferent instances of time, visible left image collector 1541determines when three new frames of visible left image data areavailable and provides those three frames as a visible image frame toblend circuit 1551. Similarly, fluorescence left image collector 1543determines when a new frame of fluorescence left image data is availableand provides that frame as a fluorescence image to blend circuit 1551Collector 1542 and collector 1544 operate similarly for the right imagesand provide the data to blend circuit 1552.

In one aspect, the capture frame rate is four times the normal videodisplay rate and so the video display has the normal number of framesper second. However, this aspect can be used with a variety of framerates. For example, the fluorescence capture frame rate could be somefraction of visible image capture frame rate. When a fluorescent frameis not captured for every RGB frame captured, the features of FIG. 9Acould be used to process the data and generate artificial fluorescenceframes, for example.

Time Division—Single Stereoscopic Optical Path with a Single CCD CameraUnit—Augmented Stereoscopic Display System 140H

In one aspect, the augmented stereoscopic video output display may beoperated in various modes. The operation of display mode select 1560,user interface 1593 and the interaction with blend circuit 1551, 1552 isthe same as the above description for display mode select 360, userinterface 393 and the interaction with blend circuit 351, 352 and thatdescription is incorporated herein by reference.

The techniques previously described for enhancing the fluorescence imagein the stereoscopic display are also applicable to this embodiment.

In all of the aspects described above, areas of fluorescence can beextracted from the images and identified with pseudo-coloring. Also,image processing to remove noise below a selected threshold may beperformed to enhance image quality.

In one embodiment the IR filter from the CCD cameras is removed toincrease sensitivity to the red and near IR wavelengths for improvedfluorescence image capture.

In any of the various video capture aspects for the visible andfluorescence images described above, the frame rate can be varied toimprove image capture. The visible images may be captured at standardvideo frame rates (e.g., 30 Hz) to provide acceptable images for thesurgeon to see the tissue and the minimally invasive surgical tools inthe stereoscopic video output at the surgeon's console. The frame ratefor the fluorescence images may be at the same frame rate used for thevisible images, or it may be slowed (e.g., 8 Hz) using the processesdescribed above. Depending upon the optical paths and the cameras unitsused, various combinations of the processes described above can be usedto generate any missing frames, either visible or fluorescence.

In some instances a slow frame rate is important to capture criticalimage information in weakly fluorescing regions. The slow frame rateallows more time for the camera/chip that is capturing the fluorescenceimages to receive the fluorescence energy from the excited fluorophoresin the tissue of interest. The movable but steady endoscopic cameraplatform provided by a robotic surgical system is a significant benefitfor the slow frame rate capture required for some fluorescence images.In contrast, a hand-held endoscope would produce low frame rate imagesthat are blurred.

In some aspects, visible images and fluorescence images captured atdifferent frame rates are synchronized in a manner similar to thatdescribed above by generating artificial fluorescence frames. In thecase of stationary cameras, such as the endoscopic camera that is heldstationary by the da Vinci® Surgical System platform, only smallrelative motions occur between camera and tissue, such as motion due tobreathing. For these small motions, blur in the captured fluorescenceimages can be ignored in many situations. In the case of moving cameras,such as when the da Vinci® Surgical System endoscopic camera is moved bythe robotic camera manipulator arm, temporal registration of visibleimages is first carried out to deblur the motion-blurred fluorescenceimages. Then, fluorescence images can be generated for imagesynchronization as described above.

As indicated above, since the fluorescence images show tissue of medicalinterest, the fluorescence images can be processed to enhance thesurgeon's video display presentation. This processing produces anartificial fluorescence image. For example, the fluorescing regions inthe fluorescence image may be artificially colored (pseudo-colored)using known methods. When the artificial fluorescence image is blendedwith the visible video image, the surgeon then sees the fluorescingtissue (e.g., artificially made bright green) in a high contrast to thesurrounding tissue in the visible image. Different image blendingoptions, such as alpha blending, of the pseudo color fluorescence imagesand visible images are made available.

In view of the above described aspects, knowledgeable persons understandthat image sensors may be positioned outside the patient at the proximalend of the endoscope, or they may be placed at the distal end of theendoscope adjacent the tissue. Left and right stereoscopic images may becaptured by separate chips or cameras, or they may be captured bydifferent regions of a single chip in a single camera.

Although alternate imaging has been described in terms of fluorescence,other imaging modalities may be included. In some aspects, the visibleimage and the alternate image may be blended in a two-dimensional(single channel; monoscopic) image capture and display system.

The seamless blending of both visible and fluorescence images inreal-time in a surgical robotic system provides a significant proceduralbenefit to the surgeon and a significant clinical benefit to thepatient. The surgeon's ability to see fluorescing tissue of interestduring surgery enhances both the precision and completeness ofidentifying and removing diseased tissue, and of identifying healthytissue that should be preserved. For example, during prostate surgerysome or all of the diseased prostate is removed, yet it is necessary topreserve adjacent nerves to avoid causing erectile dysfunction and/orurinary incontinence.

1-18. (canceled)
 19. A method of operating an image processing system ofa teleoperated system comprising: receiving a first frame including afirst visible image, the first visible image comprising a firstplurality of visible color components, the first frame being captured bya capture unit from light received by an endoscope, and the first framebeing captured at a first time; and receiving a second frame including acombination image, the combination image being a combination of a secondvisible image and a first fluorescence image, the combination imagecomprising a second plurality of color components, wherein one colorcomponent of the second plurality of color components is captured as acombination of the first fluorescence image and a visible colorcomponent, and other color component of the second plurality of colorcomponents are captured as visible color components different from thevisible color component of the one color component, the second framebeing captured by the capture unit from light received by the endoscope,and the second frame being captured at a second time, the second timebeing different from the first time; creating a second fluorescenceimage including artifacts from the first frame and the second frame;creating a third fluorescence image based on the second fluorescenceimage including artifacts; and outputting to a display system the firstvisible image and an artificial fluorescence image, the artificialfluorescence image being based on the third fluorescence image.
 20. Themethod of claim 19 further comprising: creating the artificialfluorescence image from the third fluorescence image.
 21. The method ofclaim 19, wherein the creating a second fluorescence image includingartifacts from the first frame and the second frame comprises:subtracting the first frame from the second frame to generate aplurality of frame-to-frame color component differences, wherein one ofthe frame-to-frame color component differences is the secondfluorescence image including artifacts.
 22. The method of claim 21,further comprising: normalizing the plurality of frame-to-frame colorcomponent differences.
 23. The method of claim 22, wherein the creatinga third fluorescence image based on the second fluorescence imagecomprises: performing cross-channel subtraction using the normalizedplurality of frame-to-frame color component differences to obtain thethird fluorescence image.
 24. The surgical system of claim 19, whereinthe creating a third fluorescence image based on the second fluorescenceimage comprises: performing cross-channel subtraction to obtain thethird fluorescence image.